Make repositories inaccessible to search engines?

[jd41]

The use case I am interested in (which I guess is not unprecedented) is that I want to test new stuff in a fork of an OSS project's documentation, but do not want that fork's documentation to be found online (lest people searching for documentation are confused when they hit upon my unmaintained fork in 3 years, rather than the original - another such fork already exists and is in the search index). So I am fine with my fork being public on GitHub, but not fine with the readthedocs documentation pages being in the Google/Bing/... search index. I read that there used to be an option to set repositories to "private", so they wouldn't be found with Google anymore. I guess this option would have been exactly what I wanted, but it seems to be gone. Is there an alternative today?

[jd41]

(ironically enough, I was confused for a bit searching and not finding this "private" setting in repositories - after I read about it on a PDF created on RTD of an old fork of the Readthedocs documentation, which was not updated but is still in the Google search index)

[humitos]

Read the Docs community (readthedocs.org) does not support PRIVATE versions, Read the Docs for Business (readthedocs.com) does.

However, if you just want to avoid search engines to index your documentation, you can use a robots.txt file. Take a look at https://docs.readthedocs.io/en/latest/hosting.html#custom-robots-txt-pages

I think that should be enough for your use case, right?

[jd41]

Thanks! Looking there, I see that checking "hidden" for all versions should be enough for me as well. Am I correct? Best, jd41

…

On Mon, 7 Sep 2020, 11:45 Manuel Kaufmann, ***@***.***> wrote: Read the Docs community (readthedocs.org) does not support PRIVATE versions, Read the Docs for Business (readthedocs.com) does. However, if you just want to avoid search engines to index your documentation, you can use a robots.txt file. Take a look at https://docs.readthedocs.io/en/latest/hosting.html#custom-robots-txt-pages I think that should be enough for your use case, right? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7462 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APU5ONDETJG24YL5NMH5C6TSESTTFANCNFSM4Q4AIZ6Q> . ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

[humitos]

Yes. You can mark them all as hidden and double check that the robots.txt file for your project is being generated properly by RTD.

[jd41]

The robots.txt looks okay to me now and forbids what I want to forbid ( https://nest-simulator-jd41.readthedocs.io/robots.txt), but it does point to a https://nest-simulator-jd41.readthedocs.io/sitemap.xml file, which contains the links I want to disallow. I guess this is okay?

…

On Tue, 8 Sep 2020, 11:42 Manuel Kaufmann, ***@***.***> wrote: Yes. You can mark them all as hidden and double check that the robots.txt file for your project is being generated properly by RTD. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7462 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APU5ONCIQCBWMCHUQPYMTJ3SEX4BHANCNFSM4Q4AIZ6Q> . ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

[humitos]

I guess so. I'm not sure how that affect crawlers. If that's a problem, we should open another issue and track it there. Another user already reported this at #5391 (comment). Please, subscribe there to keep updated. I'm going to close this one since the original question was answered. Thanks!