Docs: Relabel and reorganize hosting features as reference (Diátaxis) (#9952)

benjaoming · ericholscher · web-flow · commit 739fd3b07d41 · 2023-02-14T16:12:38.000+01:00
* Updated reference for hosting features, adding more features and re-organizing content * Apply suggestions from code review @ericholscher Co-authored-by: Eric Holscher <25510+ericholscher@users.noreply.github.com> * Update name of Feature reference index to avoid explicit labels elsewhere * Fix misalignments * Redo in-page references * Update "read more" link for "multiple documentation versions" * Add "private documentation" to hosting feature list * Revert to old title, modify introduction with less passive voice * Dot the list sentences, don't say "bust" but "invalidate and refresh" * Refactor of feature section * Initial refactor of features into dedicated pages. Ended up writing a bunch of additional content here, since these features were barely documented... This will be a big win for our users. * More refactoring * Remove redundancy * Remove stray Makefile commit * Make a mention of this "404/index.html" structure in relation to Sphinx DirHTML builder. * Apply suggestions from code review Co-authored-by: Benjamin Balder Bach <benjaoming@gmail.com> * Lint * Fix typo --------- Co-authored-by: Eric Holscher <25510+ericholscher@users.noreply.github.com> Co-authored-by: Eric Holscher <eric@ericholscher.com>
diff --git a/docs/user/hosting.rst b/docs/user/hosting.rst
diff --git a/docs/user/index.rst b/docs/user/index.rst
@@ -128,7 +128,6 @@ and some of the core features of Read the Docs.
   :doc:`/custom-domains` |
   :doc:`/versions` |
   :doc:`/downloadable-documentation` |
-  :doc:`/hosting` |
   :doc:`/server-side-search/index` |
   :doc:`/analytics` |
   :doc:`/pull-requests` |
@@ -159,7 +158,6 @@ and some of the core features of Read the Docs.
    /config-file/index
    /integrations
    /versions
-   /hosting
 
    /builds
    /build-customization
diff --git a/docs/user/reference/404-not-found.rst b/docs/user/reference/404-not-found.rst
@@ -0,0 +1,53 @@
+``404 Not Found`` pages
+=======================
+
+If you want your project to use a custom or branded ``404 Not Found`` page,
+you can put a ``404.html`` or ``404/index.html`` at the top level of your project's HTML output.
+
+How it works
+------------
+
+When our servers return a ``404 Not Found`` error,
+we check if there is a ``404.html`` or ``404/index.html`` in the root of your project's output.
+
+The following locations are checked, in order:
+
+* ``/404.html`` or ``404/index.html`` in the *current* documentation version.
+* ``/404.html`` or ``404/index.html`` in the  *default* documentation version.
+
+Tool integration
+----------------
+
+Documentation tools will have different ways of generating a ``404.html`` or ``404/index.html`` file.
+We have examples for some of the most popular tools below.
+
+.. tabs::
+
+   .. tab:: Sphinx
+
+      We recommend the `sphinx-notfound-page`_ extension,
+      which Read the Docs maintains.
+      It automatically creates a ``404.html`` page for your documentation,
+      matching the theme of your project.
+      See its documentation_ for how to install and customize it.
+
+      If you want to create a custom ``404.html``,
+      Sphinx uses `html_extra_path`_ option to add static files to the output.
+      You need to create a ``404.html`` file and put it under the path defined in ``html_extra_path``.
+
+      If you are using the ``DirHTML`` builder,
+      no further steps are required.
+      Sphinx will automatically apply the ``<page-name>/index.html`` folder structure to your 404 page:
+      ``404/index.html``.
+      Read the Docs also detects 404 pages named this way.
+
+   .. tab:: MkDocs
+
+      MkDocs automatically generates a ``404.html`` which Read the Docs will use.
+      However, assets will not be loaded correctly unless you define the `site_url`_ configuration value as your site's
+      :ref:`canonical base URL <canonical-urls:MkDocs>`.
+
+.. _sphinx-notfound-page: https://pypi.org/project/sphinx-notfound-page
+.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path
+.. _documentation: https://sphinx-notfound-page.readthedocs.io/
+.. _site_url: https://www.mkdocs.org/user-guide/configuration/#site_url
diff --git a/docs/user/reference/cdn.rst b/docs/user/reference/cdn.rst
@@ -0,0 +1,38 @@
+Content Delivery Network (CDN) and caching
+==========================================
+
+A CDN is used for making documentation pages *fast* for your users.
+CDNs increase speed by caching documentation content in multiple data centers around the world,
+and then serving docs from the data center closest to the user.
+
+We support CDNs on both of our sites:
+
+* On |org_brand|,
+    we are able to provide a CDN to all the projects that we host.
+    This service is graciously sponsored by `Cloudflare`_.
+* On |com_brand|,
+    the CDN is included as part of our all of our plans.
+    We use `Cloudflare`_ for this as well.
+
+CDN Benefits
+------------
+
+Having a CDN in front of your documentation has many benefits:
+
+* **Improved reliability**: Since docs are served from multiple places, one can go down and the docs are still accessible.
+* **Improved performance**: Data takes time to travel across space, so connecting to a server closer to the user makes documentation load faster.
+
+Automatic cache refresh
+-----------------------
+
+We automatically refresh the cache on the CDN when the following actions happen:
+
+* Your project is saved.
+* Your domain is saved.
+* A new version of your documentation is built.
+
+By refreshing the cache according to these rules,
+readers should **never see outdated content**.
+This makes the end-user experience seamless, and fast.
+
+.. _Cloudflare: https://www.cloudflare.com/
diff --git a/docs/user/reference/features.rst b/docs/user/reference/features.rst
@@ -1,7 +1,56 @@
-========
-Features
-========
+=================
+Feature reference
+=================
 
+.. TODO: Continue to add more features here.
+
+Hosting features
+----------------
+
+⏩️ :doc:`/custom-domains`
+  Documentation projects can use their own domain name.
+  A project may define multiple domains,
+  as well as define the *canonical domain* where all other domains will redirect.
+
+
+⏩️ :doc:`/versions`
+  We support multiple versions and translations,
+  integrated nicely into the URL of your documentation.
+  This is served at ``/en/latest/`` by default.
+  If you only have 1 version and translation,
+  we also support :doc:`single version projects </single_version>` served at ``/``.
+
+
+⏩️ :doc:`/user-defined-redirects`
+  Projects may define their own custom URL redirects,
+  with advanced functionality like folder redirects.
+
+⏩️ :doc:`/reference/cdn`
+  Documentation projects are primarily static HTML pages along with media files.
+  This allows us to cache them with our CDN,
+  making them *load faster* for your users.
+
+
+⏩️ :doc:`/reference/sitemaps`
+  Sitemaps are generated and hosted automatically,
+  improving search engine optimization.
+  This helps your users find content more effectively on your site.
+
+
+⏩️ :doc:`/reference/404-not-found`
+  A 404 page is served when we can't find a page on your site.
+  We provide a default 404 page,
+  but you can also customize it.
+
+
+⏩️ :doc:`/reference/robots`
+  `robots.txt` files allow you to customize how your documentation is indexed in search engines.
+  We provide a default robots.txt file,
+  but you can also customize it.
+
+
+⏩️ :doc:`/commercial/sharing`
+  It is possible to host private and password protected documentation on Read the Docs for Business.
 
 .. The TOC here will be refactored once we reorganize the files in docs/user/.
 .. Probably, all feature reference should be in this directory!
@@ -20,6 +69,10 @@ Features
    /automation-rules
    /user-defined-redirects
    /badges
+   /reference/cdn
+   /reference/404-not-found
+   /reference/robots
+   /reference/sitemaps
    /security-log
    /commercial/organizations
    /commercial/privacy-level
diff --git a/docs/user/reference/robots.rst b/docs/user/reference/robots.rst
@@ -0,0 +1,58 @@
+``robots.txt`` support
+======================
+
+The `robots.txt`_ files allow you to customize how your documentation is indexed in search engines.
+It's useful for:
+
+* Hiding various pages from search engines
+* Disabling certain web crawlers from accessing your documentation
+* Disallowing any indexing of your documentation
+
+Read the Docs automatically generates one for you with a configuration that works for most projects.
+By default, the automatically created ``robots.txt``:
+
+* Hides versions which are set to :ref:`versions:Hidden` from being indexed.
+* Allows indexing of all other versions.
+
+.. warning::
+
+   ``robots.txt`` files are respected by most search engines,
+   but they aren't a guarantee that your pages will not be indexed.
+   Search engines may choose to ignore your ``robots.txt`` file,
+   and index your docs anyway.
+
+   If you require *private* documentation, please see :doc:`/commercial/sharing`.
+
+How it works
+------------
+
+You can customize this file to add more rules to it.
+The ``robots.txt`` file will be served from the **default version** of your project.
+This is because the ``robots.txt`` file is served at the top-level of your domain,
+so we must choose a version to find the file in.
+The **default version** is the best place to look for it.
+
+Tool integration
+----------------
+
+Documentation tools will have different ways of generating a ``robots.txt`` file.
+We have examples for some of the most popular tools below.
+
+.. tabs::
+
+   .. tab:: Sphinx
+
+      Sphinx uses the `html_extra_path`_ configuration value to add static files to its final HTML output.
+      You need to create a ``robots.txt`` file and put it under the path defined in ``html_extra_path``.
+
+   .. tab:: MkDocs
+
+      MkDocs needs the ``robots.txt`` to be at the directory defined by the `docs_dir`_ configuration value.
+
+.. _robots.txt: https://developers.google.com/search/reference/robots_txt
+.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path
+.. _docs_dir: https://www.mkdocs.org/user-guide/configuration/#docs_dir
+
+.. seealso::
+
+   :doc:`/guides/technical-docs-seo-guide`
diff --git a/docs/user/reference/sitemaps.rst b/docs/user/reference/sitemaps.rst
