Merge branch 'master' into refactor-search-apiview

Author: stsewd

CHANGELOG.rst

@@ -1,3 +1,21 @@
+Version 5.1.1
+-------------
+
+:Date: May 26, 2020
+
+* `@stsewd <https://github.com/stsewd>`__: Search: show total_results from last query (`#7101 <https://github.com/readthedocs/readthedocs.org/pull/7101>`__)
+* `@humitos <https://github.com/humitos>`__: Add a tip in EmbedAPI to use Sphinx reference in section (`#7099 <https://github.com/readthedocs/readthedocs.org/pull/7099>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Release 5.1.0 (`#7098 <https://github.com/readthedocs/readthedocs.org/pull/7098>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Add a setting for storing pageviews (`#7097 <https://github.com/readthedocs/readthedocs.org/pull/7097>`__)
+* `@humitos <https://github.com/humitos>`__: Document Embed APIv2 endpoint (`#7095 <https://github.com/readthedocs/readthedocs.org/pull/7095>`__)
+* `@stsewd <https://github.com/stsewd>`__: Footer: Check for mkdocs doctype too (`#7094 <https://github.com/readthedocs/readthedocs.org/pull/7094>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Fix the unresolver not working properly with root paths (`#7093 <https://github.com/readthedocs/readthedocs.org/pull/7093>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Add a project-level configuration for PR builds (`#7090 <https://github.com/readthedocs/readthedocs.org/pull/7090>`__)
+* `@santos22 <https://github.com/santos22>`__: Fix tests ahead of django-dynamic-fixture update (`#7073 <https://github.com/readthedocs/readthedocs.org/pull/7073>`__)
+* `@ericholscher <https://github.com/ericholscher>`__: Add ability for users to set their own URLConf (`#6963 <https://github.com/readthedocs/readthedocs.org/pull/6963>`__)
+* `@dojutsu-user <https://github.com/dojutsu-user>`__: Store Pageviews in DB (`#6121 <https://github.com/readthedocs/readthedocs.org/pull/6121>`__)
+* `@humitos <https://github.com/humitos>`__: GitLab Integration (`#3327 <https://github.com/readthedocs/readthedocs.org/pull/3327>`__)
+
 Version 5.1.0
 -------------
 

docs/api/v2.rst

@@ -317,6 +317,85 @@ Build detail
 
     Some fields primarily used for UI elements in Read the Docs are omitted.
 
+
+Embed
+~~~~~
+
+.. http:get::  /api/v2/embed/
+
+    Retrieve HTML-formatted content from documentation page or section.
+
+    **Example request**:
+
+    .. prompt:: bash $
+
+        curl https://readthedocs.org/api/v2/embed/?project=docs&version=latest&doc=features&path=features.html
+
+    or
+
+    .. prompt:: bash $
+
+        curl https://readthedocs.org/api/v2/embed/?url=https://docs.readthedocs.io/en/latest/features.html
+
+    **Example response**:
+
+    .. sourcecode:: js
+
+        {
+            "content": [
+                "<div class=\"section\" id=\"read-the-docs-features\">\n<h1>Read the Docs..."
+            ],
+            "headers": [
+                {
+                    "Read the Docs features": "#"
+                },
+                {
+                    "Automatic Documentation Deployment": "#automatic-documentation-deployment"
+                },
+                {
+                    "Custom Domains & White Labeling": "#custom-domains-white-labeling"
+                },
+                {
+                    "Versioned Documentation": "#versioned-documentation"
+                },
+                {
+                    "Downloadable Documentation": "#downloadable-documentation"
+                },
+                {
+                    "Full-Text Search": "#full-text-search"
+                },
+                {
+                    "Open Source and Customer Focused": "#open-source-and-customer-focused"
+                }
+            ],
+            "url": "https://docs.readthedocs.io/en/latest/features",
+            "meta": {
+                "project": "docs",
+                "version": "latest",
+                "doc": "features",
+                "section": "read the docs features"
+            }
+        }
+
+    :>json string content: HTML content of the section.
+    :>json object headers: section's headers in the document.
+    :>json string url: URL of the document.
+    :>json object meta: meta data of the requested section.
+
+    :query string project: Read the Docs project's slug.
+    :query string doc: document to fetch content from.
+
+    :query string version: *optional* Read the Docs version's slug (default: ``latest``).
+    :query string section: *optional* section within the document to fetch.
+    :query string path: *optional* full path to the document including extension.
+
+    :query string url: full URL of the document (and section) to fetch content from.
+
+    .. note::
+
+       You can call this endpoint by sending at least ``project`` and ``doc`` *or* ``url`` attribute.
+
+
 Undocumented resources and endpoints
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/guides/autobuild-docs-for-pull-requests.rst

@@ -2,8 +2,11 @@ Autobuild Documentation for Pull Requests
 =========================================
 
 Read the Docs allows autobuilding documentation for pull/merge requests for GitHub or GitLab projects.
-This feature is currently available under a :doc:`Feature Flag </guides/feature-flags>`.
-So, you can enable this feature by sending us an `email <mailto:[email protected]>`__ including your project URL.
+This feature is currently enabled for a subset of our projects while being rolled out.
+You can check to see if your project has it enabled by looking at the :guilabel:`Admin > Advanced settings` and look for :guilabel:`Build pull requests for this project`.
+We are rolling this feature out based on the projects age on Read the Docs,
+so older projects will get it first.
+You can also ask for this feature by sending us an `email <mailto:[email protected]>`__ including your project URL.
 
 Features
 --------

docs/guides/embedding-content.rst

@@ -73,6 +73,32 @@ from our own docs and will populate the content of it into the ``#help-container
 You can modify this example to subscribe to ``.onclick`` Javascript event,
 and show a modal when the user clicks in a "Help" link.
 
+.. tip::
+
+    Take into account that if the title changes, your ``section`` argument will break.
+    To avoid that, you can manually define Sphinx references above the sections you don't want to break.
+    For example,
+
+    .. code-block:: rst
+       :emphasize-lines: 3
+
+       .. in your .rst document file
+
+       .. _unbreakable-section-reference:
+
+       Creating an automation rule
+       ---------------------------
+
+       This is the text of the section.
+
+    To link to the section "Creating an automation rule" you can send ``section=unbreakable-section-reference``.
+    If you change the title it won't break the embedded content because the label for that title will still be ``unbreakable-section-reference``.
+
+    Please, take a look at the Sphinx `:ref:` `role documentation`_ for more information about how to create references.
+
+    .. _role documentation: https://www.sphinx-doc.org/en/stable/usage/restructuredtext/roles.html#role-ref
+
+
 Calling the Embed API directly
 ------------------------------
 

readthedocs/api/v2/serializers.py

@@ -27,6 +27,7 @@ class Meta:
             'documentation_type',
             'users',
             'canonical_url',
+            'urlconf',
         )
 
 

readthedocs/api/v2/views/integrations.py

@@ -412,6 +412,7 @@ def handle_webhook(self):
         # Handle pull request events
         if all([
                 self.project.has_feature(Feature.EXTERNAL_VERSION_BUILD),
+                self.project.external_builds_enabled,
                 event == GITHUB_PULL_REQUEST,
                 action,
         ]):
@@ -568,6 +569,7 @@ def handle_webhook(self):
 
         if (
             self.project.has_feature(Feature.EXTERNAL_VERSION_BUILD) and
+            self.project.external_builds_enabled and
             event == GITLAB_MERGE_REQUEST and action
         ):
             if (

readthedocs/core/resolver.py

@@ -62,6 +62,7 @@ def base_resolve_path(
             subproject_slug=None,
             subdomain=None,
             cname=None,
+            urlconf=None,
     ):
         """Resolve a with nothing smart, just filling in the blanks."""
         # Only support `/docs/project' URLs outside our normal environment. Normally
@@ -79,6 +80,31 @@ def base_resolve_path(
         else:
             url += '{language}/{version_slug}/{filename}'
 
+        # Allow users to override their own URLConf
+        # This logic could be cleaned up with a standard set of variable replacements
+        if urlconf:
+            url = urlconf
+            url = url.replace(
+                '$version',
+                '{version_slug}',
+            )
+            url = url.replace(
+                '$language',
+                '{language}',
+            )
+            url = url.replace(
+                '$filename',
+                '{filename}',
+            )
+            url = url.replace(
+                '$subproject',
+                '{subproject_slug}',
+            )
+            if '$' in url:
+                log.warning(
+                    'Unconverted variable in a resolver URLConf: url=%s', url
+                )
+
         return url.format(
             project_slug=project_slug,
             filename=filename,
@@ -97,6 +123,7 @@ def resolve_path(
             single_version=None,
             subdomain=None,
             cname=None,
+            urlconf=None,
     ):
         """Resolve a URL with a subset of fields defined."""
         version_slug = version_slug or project.get_default_version()
@@ -122,6 +149,7 @@ def resolve_path(
             subproject_slug=subproject_slug,
             cname=cname,
             subdomain=subdomain,
+            urlconf=urlconf or project.urlconf,
         )
 
     def resolve_domain(self, project):

readthedocs/core/static-src/core/js/doc-embed/rtd-data.js

@@ -59,8 +59,10 @@ function get() {
 
     $.extend(config, defaults, window.READTHEDOCS_DATA);
 
-    // Force to use new settings
-    config.proxied_api_host = '/_';
+    if (!("proxied_api_host" in config)) {
+        // Use direct proxied API host
+        config.proxied_api_host = '/_';
+    }
 
     return config;
 }

readthedocs/core/static/core/js/readthedocs-doc-embed.js

@@ -234,6 +234,7 @@ def build(self):
         build_command = [
             *self.get_sphinx_cmd(),
             '-T',
+            *self.sphinx_parallel_arg(),
         ]
         if self._force:
             build_command.append('-E')
@@ -270,6 +271,11 @@ def get_sphinx_cmd(self):
             self.python_env.venv_bin(filename='sphinx-build'),
         )
 
+    def sphinx_parallel_arg(self):
+        if self.project.has_feature(Feature.SPHINX_PARALLEL):
+            return ['-j', 'auto']
+        return []
+
     def venv_sphinx_supports_latexmk(self):
         """
         Check if ``sphinx`` from the user's venv supports ``latexmk``.
@@ -448,6 +454,7 @@ def build(self):
             *self.get_sphinx_cmd(),
             '-b',
             'latex',
+            *self.sphinx_parallel_arg(),
             '-D',
             'language={lang}'.format(lang=self.project.language),
             '-d',

readthedocs/doc_builder/backends/sphinx.py

@@ -97,6 +97,7 @@ context = {
     'conf_py_path': '{{ conf_py_path }}',
     'api_host': '{{ api_host }}',
     'github_user': '{{ github_user }}',
+    'proxied_api_host': '{{ project.proxied_api_host }}',
     'github_repo': '{{ github_repo }}',
     'github_version': '{{ github_version }}',
     'display_github': {{ display_github }},

readthedocs/doc_builder/templates/doc_builder/conf.py.tmpl

@@ -203,6 +203,7 @@ class Meta:
             'analytics_code',
             'show_version_warning',
             'single_version',
+            'external_builds_enabled'
         )
         # These that can be set per-version using a config file.
         per_version_settings = (
@@ -259,6 +260,10 @@ def __init__(self, *args, **kwargs):
         else:
             self.fields['default_version'].widget.attrs['readonly'] = True
 
+        # Enable PR builder option on projects w/ feature flag
+        if not self.instance.has_feature(Feature.EXTERNAL_VERSION_BUILD):
+            self.fields.pop('external_builds_enabled')
+
     def clean_conf_py_file(self):
         filename = self.cleaned_data.get('conf_py_file', '').strip()
         if filename and 'conf.py' not in filename:

readthedocs/projects/forms.py

@@ -0,0 +1,18 @@
+# Generated by Django 2.2.10 on 2020-05-21 22:18
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('projects', '0048_remove_version_privacy_field'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='project',
+            name='external_builds_enabled',
+            field=models.BooleanField(default=False, help_text='More information in <a href="https://docs.readthedocs.io/en/latest/guides/autobuild-docs-for-pull-requests.html">our docs</a>', null=True, verbose_name='Build pull requests for this project'),
+        ),
+    ]

readthedocs/projects/migrations/0049_add_external_build_enabled.py

@@ -0,0 +1,22 @@
+# Generated by Django 2.2.10 on 2020-05-19 13:24
+
+from django.db import migrations
+
+
+def migrate_features(apps, schema_editor):
+    # Enable the PR builder for projects with the feature flag
+    Feature = apps.get_model('projects', 'Feature')
+    if Feature.objects.filter(feature_id='external_version_build').exists():
+        for project in Feature.objects.get(feature_id='external_version_build').projects.all():
+            project.external_builds_enabled = True
+            project.save()
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('projects', '0049_add_external_build_enabled'),
+    ]
+
+    operations = [
+        migrations.RunPython(migrate_features),
+    ]

readthedocs/projects/migrations/0050_migrate_external_builds.py

@@ -0,0 +1,18 @@
+# Generated by Django 2.2.10 on 2020-05-26 14:34
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('projects', '0050_migrate_external_builds'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='project',
+            name='urlconf',
+            field=models.CharField(default=None, help_text='Supports the following keys: $language, $version, $subproject, $filename. An example: `$language/$version/$filename`.', max_length=255, null=True, verbose_name='Documentation URL Configuration'),
+        ),
+    ]