Search: custom search page ranking (#7237)

* Add search page rankings

* Update docs

* Fix tests

* Fix linter

* Update docs

* Pass dict, config object isn't serializable

* Do ranking at query time

* Update docs

* Linter

* Add tip

* Add comment

* TODO about alternative precedence

Author: stsewd

docs/config-file/v2.rst

@@ -25,7 +25,7 @@ Below is an example YAML file which shows the most common configuration options:
     #  configuration: mkdocs.yml
 
     # Optionally build your docs in additional formats such as PDF
-    formats: 
+    formats:
       - pdf
 
     # Optionally set the version of Python and requirements required to build your docs
@@ -74,11 +74,15 @@ Example:
 
 .. code-block:: yaml
 
+   version: 2
+
    # Default
    formats: []
 
 .. code-block:: yaml
 
+   version: 2
+
    # Build PDF & ePub
    formats:
      - epub
@@ -90,6 +94,8 @@ Example:
 
    .. code-block:: yaml
 
+      version: 2
+
       # Build all formats
       formats: all
 
@@ -104,6 +110,8 @@ Configuration of the Python environment to be used.
 
 .. code-block:: yaml
 
+   version: 2
+
    python:
       version: 3.7
       install:
@@ -153,6 +161,8 @@ Example:
 
 .. code-block:: yaml
 
+   version: 2
+
    python:
       version: 3.7
       install:
@@ -199,6 +209,8 @@ Example:
 
 .. code-block:: yaml
 
+   version: 2
+
    python:
       version: 3.7
       install:
@@ -242,6 +254,8 @@ Configuration for Conda support.
 
 .. code-block:: yaml
 
+   version: 2
+
    conda:
      environment: environment.yml
 
@@ -260,6 +274,8 @@ Configuration for the documentation build process.
 
 .. code-block:: yaml
 
+   version: 2
+
    build:
      image: latest
 
@@ -290,6 +306,8 @@ Configuration for Sphinx documentation
 
 .. code-block:: yaml
 
+   version: 2
+
    sphinx:
      builder: html
      configuration: conf.py
@@ -335,6 +353,8 @@ Configuration for Mkdocs documentation.
 
 .. code-block:: yaml
 
+   version: 2
+
    mkdocs:
      configuration: mkdocs.yml
      fail_on_warning: false
@@ -374,6 +394,8 @@ VCS submodules configuration.
 
 .. code-block:: yaml
 
+   version: 2
+
    submodules:
      include:
         - one
@@ -394,6 +416,8 @@ List of submodules to be included.
 
    .. code-block:: yaml
 
+      version: 2
+
       submodules:
         include: all
 
@@ -412,6 +436,8 @@ List of submodules to be excluded.
 
    .. code-block:: yaml
 
+      version: 2
+
       submodules:
         exclude: all
 
@@ -427,6 +453,65 @@ Do a recursive clone of the submodules.
 
    This is ignored if there aren't submodules to clone.
 
+search
+~~~~~~
+
+Settings for more control over :doc:`/server-side-search`.
+
+.. code-block:: yaml
+
+   version: 2
+
+   search:
+      ranking:
+        api/v1/*: -1
+        api/v2/*: 4
+
+search.ranking
+``````````````
+
+Set a custom search rank over pages matching a pattern.
+
+:Type: ``map`` of patterns to ranks
+:Default: ``{}``
+
+Patterns are matched against the final html pages produced by the build
+(you should try to match `index.html`, not `docs/index.rst`).
+Patterns can include some special characters:
+
+- ``*`` matches everything
+- ``?`` matches any single character
+- ``[seq]`` matches any character in ``seq``
+
+The rank can be an integer number between -10 and 10 (inclusive).
+Pages with a rank closer to -10 will appear further down the list of results,
+and pages with a rank closer to 10 will appear higher in the list of results.
+Note that 0 means *normal rank*, not *no rank*.
+
+.. code-block:: yaml
+
+   version: 2
+
+   search:
+      ranking:
+        # Match a single file
+        tutorial.hml: 2
+
+        # Match all files under the api/v1 directory
+        api/v1/*: -5
+
+        # Match all files that end with tutorial.html
+        */tutorial.html: 3
+
+.. note::
+
+   The final rank will be the last pattern to match the page.
+
+.. tip::
+
+   Is better to decrease the rank of pages you want to deprecate,
+   rather than increasing the rank of the other pages.
+
 Schema
 ------
 
@@ -468,6 +553,7 @@ New settings
 - :ref:`config-file/v2:mkdocs`
 - :ref:`config-file/v2:submodules`
 - :ref:`config-file/v2:python.install`
+- :ref:`config-file/v2:search`
 
 Migrating from the web interface
 --------------------------------

docs/guides/searching-with-readthedocs.rst

@@ -2,7 +2,7 @@ Searching with Read the Docs
 ============================
 
 Read the Docs uses :doc:`/server-side-search` to power our search.
-This guide explains how to add a "search as you type" feature to your documentation, 
+This guide explains how to add a "search as you type" feature to your documentation,
 and how to use advanced query syntax to get more accurate results.
 
 You can find information on the search architecture and how we index documents in our
@@ -20,7 +20,7 @@ Enable "search as you type" in your documentation
 documentation more closely with the search implementation of Read the Docs.
 It adds a clean and minimal full-page search UI that supports a **search as you type** feature.
 
-To try this feature, 
+To try this feature,
 you can press :guilabel:`/` (forward slash) and start typing or just visit these URLs:
 
 - https://docs.readthedocs.io/?rtd_search=contributing

docs/server-side-search.rst

@@ -25,7 +25,12 @@ Search results land on the exact content you were looking for
    We index every heading in the document,
    allowing you to get search results exactly to the content that you are searching for.
    Try this out by searching for `"full-text search"`_.
-   
+
+Full control over which results should be listed first
+   Set a custom rank per page,
+   allowing you to deprecate content, and always show relevant content to your users first.
+   See :ref:`config-file/v2:search.ranking`.
+
 Search across projects you have access to (|com_brand|)
    This allows you to search across all the projects you access to in your Dashboard.
    **Don't remember where you found that document the other day?
@@ -35,10 +40,10 @@ Special query syntax for more specific results.
    We support a full range of search queries.
    You can see some examples in our :ref:`guides/searching-with-readthedocs:search query syntax` guide.
 
-.. 
+..
    Code object searching
       With the user of :doc:`Sphinx Domains <sphinx:/usage/restructuredtext/domains>` we are able to automatically provide direct search results to your Code objects.
-      You can try this out with our docs here by searching for 
+      You can try this out with our docs here by searching for
       TODO: Find good examples in our docs, API maybe?
 
 .. _"full-text search": https://docs.readthedocs.io/en/latest/search.html?q=%22full-text+search%22

readthedocs/config/config.py

@@ -10,7 +10,6 @@
 from django.conf import settings
 
 from readthedocs.config.utils import list_to_dict, to_dict
-from readthedocs.projects.constants import DOCUMENTATION_CHOICES
 
 from .find import find_one
 from .models import (
@@ -20,6 +19,7 @@
     Python,
     PythonInstall,
     PythonInstallRequirements,
+    Search,
     Sphinx,
     Submodules,
 )
@@ -32,6 +32,7 @@
     validate_dict,
     validate_list,
     validate_path,
+    validate_path_pattern,
     validate_string,
 )
 
@@ -155,6 +156,7 @@ class BuildConfigBase:
         'sphinx',
         'mkdocs',
         'submodules',
+        'search',
     ]
 
     default_build_image = settings.DOCKER_DEFAULT_VERSION
@@ -652,6 +654,10 @@ def submodules(self):
             recursive=True,
         )
 
+    @property
+    def search(self):
+        return Search(ranking={})
+
 
 class BuildConfigV2(BuildConfigBase):
 
@@ -666,7 +672,6 @@ class BuildConfigV2(BuildConfigBase):
         'dirhtml': 'sphinx_htmldir',
         'singlehtml': 'sphinx_singlehtml',
     }
-    builders_display = dict(DOCUMENTATION_CHOICES)
 
     def validate(self):
         """
@@ -686,6 +691,7 @@ def validate(self):
         self._config['mkdocs'] = self.validate_mkdocs()
         self._config['sphinx'] = self.validate_sphinx()
         self._config['submodules'] = self.validate_submodules()
+        self._config['search'] = self.validate_search()
         self.validate_keys()
 
     def validate_formats(self):
@@ -1013,6 +1019,35 @@ def validate_submodules(self):
 
         return submodules
 
+    def validate_search(self):
+        """
+        Validates the search key.
+
+        - Ranking is a map of path patterns to a rank.
+        - The path pattern supports basic globs (*, ?, [seq]).
+        - The rank can be a integer number between -10 and 10.
+        """
+        raw_search = self._raw_config.get('search', {})
+        with self.catch_validation_error('search'):
+            validate_dict(raw_search)
+
+        search = {}
+        with self.catch_validation_error('search.ranking'):
+            ranking = self.pop_config('search.ranking', {})
+            validate_dict(ranking)
+
+            valid_rank_range = list(range(-10, 10 + 1))
+
+            final_ranking = {}
+            for pattern, rank in ranking.items():
+                pattern = validate_path_pattern(pattern)
+                validate_choice(rank, valid_rank_range)
+                final_ranking[pattern] = rank
+
+            search['ranking'] = final_ranking
+
+        return search
+
     def validate_keys(self):
         """
         Checks that we don't have extra keys (invalid ones).
@@ -1107,6 +1142,10 @@ def doctype(self):
     def submodules(self):
         return Submodules(**self._config['submodules'])
 
+    @property
+    def search(self):
+        return Search(**self._config['search'])
+
 
 def load(path, env_config):
     """

readthedocs/config/models.py

@@ -68,3 +68,8 @@ class Mkdocs(Base):
 class Submodules(Base):
 
     __slots__ = ('include', 'exclude', 'recursive')
+
+
+class Search(Base):
+
+    __slots__ = ('ranking',)