diff --git a/docs/user/hosting.rst b/docs/user/hosting.rst deleted file mode 100644 index 823c2594661..00000000000 --- a/docs/user/hosting.rst +++ /dev/null @@ -1,130 +0,0 @@ -Documentation Hosting Features -============================== - -The main way that users interact with your documentation is via the hosted HTML that we serve. -We support a number of important features that you would expect for a documentation host. - -.. contents:: Contents - :local: - :depth: 1 - -Subdomain support ------------------ - -Every project has a subdomain that is available to serve its documentation based on it's :term:`slug`. -If you go to ``.readthedocs.io``, it should show you the latest version of your documentation, -for example https://docs.readthedocs.io. -For :doc:`/commercial/index` the subdomain looks like ``.readthedocs-hosted.com``. - -.. seealso:: - - :doc:`/custom-domains` - More information on using a custom domain. - -Content Delivery Network (CDN) ------------------------------- - -A CDN is used for making documentation pages faster for your users. -This is done by caching the documentation page content in multiple data centers around the world, -and then serving docs from the data center closest to the user. - -We support CDNs on both of our sites: - -.. tabs:: - - .. tab:: |org_brand| - - On |org_brand|, - we are able to provide a CDN to all the projects that we host. - This service is graciously sponsored by `Cloudflare`_. - - We bust the cache on the CDN when the following actions happen: - - * Your Project is saved - * Your Domain is saved - * A new version is built - - - .. tab:: |com_brand| - - On |com_brand|, - we offer a CDN as part of our **Pro plan** and above. - Please contact support@readthedocs.com to discuss how we can enable this for you. - - We invalidate and refresh the cache on the CDN when the following actions happen: - - * Your project is saved - * Your domain is saved - * A version or branch is built - -.. _Cloudflare: https://www.cloudflare.com/ - -Sitemaps --------- - -`Sitemaps `__ allows us to inform search engines about URLs that are available for crawling -and communicate them additional information about each URL of the project: - -* when it was last updated, -* how often it changes, -* how important it is in relation to other URLs in the site, and -* what translations are available for a page. - -Read the Docs automatically generates a sitemap for each project that hosts -to improve results when performing a search on these search engines. -This allow us to prioritize results based on the version number, for example -to show ``stable`` as the top result followed by ``latest`` and then all the project's -versions sorted following `semantic versioning`_. - -.. _semantic versioning: https://semver.org/ - -Custom Not Found (404) Pages ----------------------------- - -If you want your project to use a custom page for not found pages instead of the "Maze Found" default, -you can put a ``404.html`` or ``404/index.html`` in your project's HTML output. - -When a 404 is returned, -Read the Docs checks if there is a ``404.html`` or ``404/index.html`` in your project's output -corresponding to the *current* version -and uses it if it exists. -Otherwise, it tries to fall back to the ``404.html`` or ``404/index.html`` page -corresponding to the *default* version of the project. - -We recommend the `sphinx-notfound-page`_ extension, -which Read the Docs maintains. -It automatically creates a the 404 page for your documentation, -matching the theme of your project. -See its documentation_ for how to install and customize it. - -.. _sphinx-notfound-page: https://pypi.org/project/sphinx-notfound-page -.. _documentation: https://sphinx-notfound-page.readthedocs.io/ - -Custom robots.txt Pages ------------------------ - -`robots.txt`_ files allow you to customize how your documentation is indexed in search engines. -We automatically generate one for you, -which automatically hides versions which are set to :ref:`versions:Hidden`. - -The ``robots.txt`` file will be served from the **default version** of your Project. -This is because the ``robots.txt`` file is served at the top-level of your domain, -so we must choose a version to find the file in. -The **default version** is the best place to look for it. - -Sphinx and Mkdocs both have different ways of outputting static files in the build: - -Sphinx -~~~~~~ - -Sphinx uses `html_extra_path`_ option to add static files to the output. -You need to create a ``robots.txt`` file and put it under the path defined in ``html_extra_path``. - -MkDocs -~~~~~~ - -MkDocs needs the ``robots.txt`` to be at the directory defined at `docs_dir`_ config. - -.. _robots.txt: https://developers.google.com/search/reference/robots_txt -.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path -.. _docs_dir: https://www.mkdocs.org/user-guide/configuration/#docs_dir diff --git a/docs/user/index.rst b/docs/user/index.rst index 6bfb5983b33..d883521f5c1 100644 --- a/docs/user/index.rst +++ b/docs/user/index.rst @@ -128,7 +128,6 @@ and some of the core features of Read the Docs. :doc:`/custom-domains` | :doc:`/versions` | :doc:`/downloadable-documentation` | - :doc:`/hosting` | :doc:`/server-side-search/index` | :doc:`/analytics` | :doc:`/pull-requests` | @@ -159,7 +158,6 @@ and some of the core features of Read the Docs. /config-file/index /integrations /versions - /hosting /builds /build-customization diff --git a/docs/user/reference/404-not-found.rst b/docs/user/reference/404-not-found.rst new file mode 100644 index 00000000000..1cfba4ba00c --- /dev/null +++ b/docs/user/reference/404-not-found.rst @@ -0,0 +1,53 @@ +``404 Not Found`` pages +======================= + +If you want your project to use a custom or branded ``404 Not Found`` page, +you can put a ``404.html`` or ``404/index.html`` at the top level of your project's HTML output. + +How it works +------------ + +When our servers return a ``404 Not Found`` error, +we check if there is a ``404.html`` or ``404/index.html`` in the root of your project's output. + +The following locations are checked, in order: + +* ``/404.html`` or ``404/index.html`` in the *current* documentation version. +* ``/404.html`` or ``404/index.html`` in the *default* documentation version. + +Tool integration +---------------- + +Documentation tools will have different ways of generating a ``404.html`` or ``404/index.html`` file. +We have examples for some of the most popular tools below. + +.. tabs:: + + .. tab:: Sphinx + + We recommend the `sphinx-notfound-page`_ extension, + which Read the Docs maintains. + It automatically creates a ``404.html`` page for your documentation, + matching the theme of your project. + See its documentation_ for how to install and customize it. + + If you want to create a custom ``404.html``, + Sphinx uses `html_extra_path`_ option to add static files to the output. + You need to create a ``404.html`` file and put it under the path defined in ``html_extra_path``. + + If you are using the ``DirHTML`` builder, + no further steps are required. + Sphinx will automatically apply the ``/index.html`` folder structure to your 404 page: + ``404/index.html``. + Read the Docs also detects 404 pages named this way. + + .. tab:: MkDocs + + MkDocs automatically generates a ``404.html`` which Read the Docs will use. + However, assets will not be loaded correctly unless you define the `site_url`_ configuration value as your site's + :ref:`canonical base URL `. + +.. _sphinx-notfound-page: https://pypi.org/project/sphinx-notfound-page +.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path +.. _documentation: https://sphinx-notfound-page.readthedocs.io/ +.. _site_url: https://www.mkdocs.org/user-guide/configuration/#site_url diff --git a/docs/user/reference/cdn.rst b/docs/user/reference/cdn.rst new file mode 100644 index 00000000000..e7ed31e589b --- /dev/null +++ b/docs/user/reference/cdn.rst @@ -0,0 +1,38 @@ +Content Delivery Network (CDN) and caching +========================================== + +A CDN is used for making documentation pages *fast* for your users. +CDNs increase speed by caching documentation content in multiple data centers around the world, +and then serving docs from the data center closest to the user. + +We support CDNs on both of our sites: + +* On |org_brand|, + we are able to provide a CDN to all the projects that we host. + This service is graciously sponsored by `Cloudflare`_. +* On |com_brand|, + the CDN is included as part of our all of our plans. + We use `Cloudflare`_ for this as well. + +CDN Benefits +------------ + +Having a CDN in front of your documentation has many benefits: + +* **Improved reliability**: Since docs are served from multiple places, one can go down and the docs are still accessible. +* **Improved performance**: Data takes time to travel across space, so connecting to a server closer to the user makes documentation load faster. + +Automatic cache refresh +----------------------- + +We automatically refresh the cache on the CDN when the following actions happen: + +* Your project is saved. +* Your domain is saved. +* A new version of your documentation is built. + +By refreshing the cache according to these rules, +readers should **never see outdated content**. +This makes the end-user experience seamless, and fast. + +.. _Cloudflare: https://www.cloudflare.com/ diff --git a/docs/user/reference/features.rst b/docs/user/reference/features.rst index 2fd3f7df59c..75265978663 100644 --- a/docs/user/reference/features.rst +++ b/docs/user/reference/features.rst @@ -1,7 +1,56 @@ -======== -Features -======== +================= +Feature reference +================= +.. TODO: Continue to add more features here. + +Hosting features +---------------- + +⏩️ :doc:`/custom-domains` + Documentation projects can use their own domain name. + A project may define multiple domains, + as well as define the *canonical domain* where all other domains will redirect. + + +⏩️ :doc:`/versions` + We support multiple versions and translations, + integrated nicely into the URL of your documentation. + This is served at ``/en/latest/`` by default. + If you only have 1 version and translation, + we also support :doc:`single version projects ` served at ``/``. + + +⏩️ :doc:`/user-defined-redirects` + Projects may define their own custom URL redirects, + with advanced functionality like folder redirects. + +⏩️ :doc:`/reference/cdn` + Documentation projects are primarily static HTML pages along with media files. + This allows us to cache them with our CDN, + making them *load faster* for your users. + + +⏩️ :doc:`/reference/sitemaps` + Sitemaps are generated and hosted automatically, + improving search engine optimization. + This helps your users find content more effectively on your site. + + +⏩️ :doc:`/reference/404-not-found` + A 404 page is served when we can't find a page on your site. + We provide a default 404 page, + but you can also customize it. + + +⏩️ :doc:`/reference/robots` + `robots.txt` files allow you to customize how your documentation is indexed in search engines. + We provide a default robots.txt file, + but you can also customize it. + + +⏩️ :doc:`/commercial/sharing` + It is possible to host private and password protected documentation on Read the Docs for Business. .. The TOC here will be refactored once we reorganize the files in docs/user/. .. Probably, all feature reference should be in this directory! @@ -20,6 +69,10 @@ Features /automation-rules /user-defined-redirects /badges + /reference/cdn + /reference/404-not-found + /reference/robots + /reference/sitemaps /security-log /commercial/organizations /commercial/privacy-level diff --git a/docs/user/reference/robots.rst b/docs/user/reference/robots.rst new file mode 100644 index 00000000000..43c667310c8 --- /dev/null +++ b/docs/user/reference/robots.rst @@ -0,0 +1,58 @@ +``robots.txt`` support +====================== + +The `robots.txt`_ files allow you to customize how your documentation is indexed in search engines. +It's useful for: + +* Hiding various pages from search engines +* Disabling certain web crawlers from accessing your documentation +* Disallowing any indexing of your documentation + +Read the Docs automatically generates one for you with a configuration that works for most projects. +By default, the automatically created ``robots.txt``: + +* Hides versions which are set to :ref:`versions:Hidden` from being indexed. +* Allows indexing of all other versions. + +.. warning:: + + ``robots.txt`` files are respected by most search engines, + but they aren't a guarantee that your pages will not be indexed. + Search engines may choose to ignore your ``robots.txt`` file, + and index your docs anyway. + + If you require *private* documentation, please see :doc:`/commercial/sharing`. + +How it works +------------ + +You can customize this file to add more rules to it. +The ``robots.txt`` file will be served from the **default version** of your project. +This is because the ``robots.txt`` file is served at the top-level of your domain, +so we must choose a version to find the file in. +The **default version** is the best place to look for it. + +Tool integration +---------------- + +Documentation tools will have different ways of generating a ``robots.txt`` file. +We have examples for some of the most popular tools below. + +.. tabs:: + + .. tab:: Sphinx + + Sphinx uses the `html_extra_path`_ configuration value to add static files to its final HTML output. + You need to create a ``robots.txt`` file and put it under the path defined in ``html_extra_path``. + + .. tab:: MkDocs + + MkDocs needs the ``robots.txt`` to be at the directory defined by the `docs_dir`_ configuration value. + +.. _robots.txt: https://developers.google.com/search/reference/robots_txt +.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path +.. _docs_dir: https://www.mkdocs.org/user-guide/configuration/#docs_dir + +.. seealso:: + + :doc:`/guides/technical-docs-seo-guide` diff --git a/docs/user/reference/sitemaps.rst b/docs/user/reference/sitemaps.rst new file mode 100644 index 00000000000..49483613a92 --- /dev/null +++ b/docs/user/reference/sitemaps.rst @@ -0,0 +1,35 @@ +``sitemap.xml`` support +======================= + +`Sitemaps `__ allow you to inform search engines about URLs that are available for crawling. +This makes your content more :term:`discoverable `, +and improves your :doc:`Search Engine Optimization (SEO) `. + +How it works +------------ + +The ``sitemap.xml`` file is read by search engines in order to index your documentation. +It contains information such as: + +* When a URL was last updated. +* How often that URL changes. +* How important this URL is in relation to other URLs in the site. +* What translations are available for a page. + +Read the Docs automatically generates a ``sitemap.xml`` for your project, + +By default the sitemap includes: + +* Each version of your documentation and when it was last updated, sorted by version number. + +This allows search engines to prioritize results based on the version number, +sorted by `semantic versioning`_. + +Custom ``sitemap.xml`` +---------------------- + +If you need a custom sitemap, +please let us know by contacting :doc:`/support`. + +.. _semantic versioning: https://semver.org/ +.. _GitHub issue #5391: https://github.com/readthedocs/readthedocs.org/issues/5391