Docs: Relabel and reorganize hosting features as reference (Diátaxis)

docs/user/index.rst

@@ -128,7 +128,6 @@ and some of the core features of Read the Docs.
   :doc:`/custom-domains` |
   :doc:`/versions` |
   :doc:`/downloadable-documentation` |
-  :doc:`/hosting` |
   :doc:`/server-side-search/index` |
   :doc:`/analytics` |
   :doc:`/pull-requests` |
@@ -159,7 +158,6 @@ and some of the core features of Read the Docs.
    /config-file/index
    /integrations
    /versions
-   /hosting
 
    /builds
    /build-customization

docs/user/reference/404-not-found.rst

@@ -0,0 +1,53 @@
+``404 Not Found`` pages
+=======================
+
+If you want your project to use a custom or branded ``404 Not Found`` page,
+you can put a ``404.html`` or ``404/index.html`` at the top level of your project's HTML output.
+
+How it works
+------------
+
+When our servers return a ``404 Not Found`` error,
+we check if there is a ``404.html`` or ``404/index.html`` in the root of your project's output.
+
+The following locations are checked, in order:
+
+* ``/404.html`` or ``404/index.html`` in the *current* documentation version.
+* ``/404.html`` or ``404/index.html`` in the  *default* documentation version.
+
+Tool Integration
+----------------
+
+Documentation tools will have different ways of generating a ``404.html`` or ``404/index.html`` file.
+We have examples for some of the most popular tools below.
+
+.. tabs::
+
+   .. tab:: Sphinx
+
+      We recommend the `sphinx-notfound-page`_ extension,
+      which Read the Docs maintains.
+      It automatically creates a ``404.html`` page for your documentation,
+      matching the theme of your project.
+      See its documentation_ for how to install and customize it.
+
+      If you want to create a custom ``404.html``,
+      Sphinx uses `html_extra_path`_ option to add static files to the output.
+      You need to create a ``404.html`` file and put it under the path defined in ``html_extra_path``.
+
+      If you are using the ``DirHTML`` builder,
+      no further steps are required.
+      Sphinx will automatically apply the ``<page-name>/index.html`` folder structure to your 404 page:
+      ``404/index.html``.
+      Read the Docs also detects 404 pages named this way.
+
+   .. tab:: MkDocs
+
+      MkDocs automatically generates a ``404.html`` which Read the Docs will use.
+      However, assets will not be loaded correctly unless you define the `site_url`_ configuration value as your site's
+      :ref:`canonical base URL <canonical-urls:MkDocs>`.
+
+.. _sphinx-notfound-page: https://pypi.org/project/sphinx-notfound-page
+.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path
+.. _documentation: https://sphinx-notfound-page.readthedocs.io/
+.. _site_url: https://www.mkdocs.org/user-guide/configuration/#site_url

docs/user/reference/cdn.rst

@@ -0,0 +1,39 @@
+Content Delivery Network (CDN) and caching
+==========================================
+
+A CDN is used for making documentation pages *fast* for your users.
+This is done by caching the documentation page content in multiple data centers around the world,
+and then serving docs from the data center closest to the user.
+
+We support CDNs on both of our sites:
+
+* On |org_brand|,
+    we are able to provide a CDN to all the projects that we host.
+    This service is graciously sponsored by `Cloudflare`_.
+* On |com_brand|,
+    the CDN is included as part of our all of our plans.
+    We use `Cloudflare`_ for this as well.
+
+CDN Benefits
+------------
+
+Having a CDN in front of your documentation has many benefits:
+
+* **Improved reliability**: Since docs are served from multiple places, one can go down and the docs are still accessible.
+* **Improved performance**: Data takes time to travel across space, so connecting to a server closer to the user makes documentation load faster.
+
+CDN Features
+------------
+
+Our integration between building and hosting documentation allows to do many smart things to make the experience seemless.
+
+We automatically refresh the cache on the CDN when the following actions happen:
+
+* Your project is saved.
+* Your domain is saved.
+* A new version of your documentation is built.
+
+This means that you should **never see outdated content**,
+but it's always as fast a possible for your users.
+
+.. _Cloudflare: https://www.cloudflare.com/

docs/user/reference/features.rst

@@ -1,7 +1,56 @@
-========
-Features
-========
+=================
+Feature reference
+=================
 
+.. TODO: Continue to add more features here.
+
+Hosting features
+----------------
+
+⏩️ :doc:`/custom-domains`
+  Documentation projects can use their own domain name.
+  A project may define multiple domains,
+  as well as define the *canonical domain* where all other domains will redirect.
+
+
+⏩️ :doc:`/versions`
+  We support multiple versions and translations,
+  integrated nicely into the URL of your documentation.
+  This is served at ``/en/latest/`` by default.
+  If you only have 1 version and translation,
+  we also support :doc:`single version projects </single_version>` served at ``/``.
+
+
+⏩️ :doc:`/user-defined-redirects`
+  Projects may define their own custom URL redirects,
+  with advanced functionality like folder redirects.
+
+⬇️ :doc:`/reference/cdn`
+  Documentation projects are primarily static HTML pages along with media files.
+  This allows us to cache them with our CDN,
+  making them *load faster* for your users.
+
+
+⬇️ :doc:`/reference/sitemaps`
+  Sitemaps are generated and hosted automatically,
+  improving search engine optimization.
+  This helps your users find content more effectively on your site.
+
+
+⬇️ :doc:`/reference/404-not-found`
+  A 404 page is served when we can't find a page on your site.
+  We provide a default 404 page,
+  but you can also customize it.
+
+
+⬇️ :doc:`/reference/robots`
+  `robots.txt` files allow you to customize how your documentation is indexed in search engines.
+  We provide a default robots.txt file,
+  but you can also customize it.
+
+
+⬇️ :doc:`/reference/robots`
+  It is possible to host private and password protected documentation on Read the Docs for Business.
 
 .. The TOC here will be refactored once we reorganize the files in docs/user/.
 .. Probably, all feature reference should be in this directory!
@@ -20,6 +69,10 @@ Features
    /automation-rules
    /user-defined-redirects
    /badges
+   /reference/cdn
+   /reference/404-not-found
+   /reference/robots
+   /reference/sitemaps
    /security-log
    /commercial/organizations
    /commercial/privacy-level

docs/user/reference/robots.rst

@@ -0,0 +1,52 @@
+``robots.txt`` support
+======================
+
+The `robots.txt`_ files allow you to customize how your documentation is indexed in search engines.
+It's useful for:
+
+* Hiding various pages from search engines
+* Disabling certain web crawlers from accessing your documentation
+* Disallowing any indexing of your documentation
+
+Read the Docs automatically generates one for you with a configuration that works for most projects.
+By default, the automatically created ``robots.txt``:abbr:
+
+* Hides versions which are set to :ref:`versions:Hidden` from being indexed.
+* Allows indexing of all other versions.
+
+.. warning:: ``robots.txt`` files are respected by search engines, but they are not a guarantee that your documentation will not be indexed.
+   Search engines may choose to ignore your ``robots.txt`` file and index your documentation anyway.
+
+How it works
+------------
+
+You can customize this file to add more rules to it.
+The ``robots.txt`` file will be served from the **default version** of your project.
+This is because the ``robots.txt`` file is served at the top-level of your domain,
+so we must choose a version to find the file in.
+The **default version** is the best place to look for it.
+
+Tool Integration
+----------------
+
+Documentation tools will have different ways of generating a ``robots.txt`` file.
+We have examples for some of the most popular tools below.
+
+.. tabs::
+
+   .. tab:: Sphinx
+
+      Sphinx uses the `html_extra_path`_ configuration value to add static files to its final HTML output.
+      You need to create a ``robots.txt`` file and put it under the path defined in ``html_extra_path``.
+
+   .. tab:: MkDocs
+
+      MkDocs needs the ``robots.txt`` to be at the directory defined by the `docs_dir`_ configuration value.
+
+.. _robots.txt: https://developers.google.com/search/reference/robots_txt
+.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path
+.. _docs_dir: https://www.mkdocs.org/user-guide/configuration/#docs_dir
+
+.. seealso::
+
+   :doc:`/guides/technical-docs-seo-guide`