Support delisting of projects

[benjaoming]

Adds support to delist projects that are unmaintained and unofficial, per policy: https://docs.readthedocs.io/en/stable/unofficial-projects.html

• Add new field Project.delisted
• Remove projects from search engine indexing (robots.txt)
• Remove from ElasticSearch indexing
• Show "delisted" status in Dashboard

Fixes https://github.com/readthedocs/meta/issues/99

[ericholscher]

This is a great start! It's pretty close with a couple tweaks, and a test 👍

[humitos]

Looks pretty close to me 👍🏼 . Good work!

[ericholscher]

We should definitely get it in next week if we can. Let's drop all the search changes here, and move that discussion here: https://github.com/readthedocs/readthedocs-ext/issues/461#issuecomment-1444063694

[benjaoming]

We've resolved issues around ES indexing for now by pushing them to https://github.com/readthedocs/readthedocs-ext/issues/461

There isn't any other outstanding items here, so would be great with a review (and possibly merge) @humitos or @stsewd ?

Btw bumped into an interesting use-case today where someone wanted to opt out of in-doc search, but likely not global/dashboard search. We'll probably be back at defining what "delisted" means, and maybe even expose it in the UI - but I think that it's okay to say that "delisted" is currently a general option and everything else will be easy to extend on top of this.

[ericholscher]

I'm going to merge this because I was hoping to ship it this week 👍

[benjaoming]

great @ericholscher 👍 I was aiming for someone in an earlier timezone to make it before release :)