Merge remote-tracking branch 'origin/master' into davidfischer/storage-epubs-pdfs-zips

Author: ericholscher

CHANGELOG.rst

@@ -1,3 +1,18 @@
+Version 3.2.3
+-------------
+
+:Date: February 19, 2019
+
+* `@ericholscher <http://github.com/ericholscher>`__: Add basic auth to the generic webhook API. (`#5311 <https://github.com/rtfd/readthedocs.org/pull/5311>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Fix an issue where we were not properly filtering projects (`#5309 <https://github.com/rtfd/readthedocs.org/pull/5309>`__)
+* `@stsewd <http://github.com/stsewd>`__: Rstrip repo url (`#5308 <https://github.com/rtfd/readthedocs.org/pull/5308>`__)
+* `@stsewd <http://github.com/stsewd>`__: Use autosectionlabel for docs in security (`#5307 <https://github.com/rtfd/readthedocs.org/pull/5307>`__)
+* `@rexzing <http://github.com/rexzing>`__: Incompatible dependency for prospector with pylint-django (`#5306 <https://github.com/rtfd/readthedocs.org/pull/5306>`__)
+* `@pyup-bot <http://github.com/pyup-bot>`__: pyup:  Scheduled weekly dependency update for week 07 (`#5305 <https://github.com/rtfd/readthedocs.org/pull/5305>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Allow extensions to control URL structure (`#5296 <https://github.com/rtfd/readthedocs.org/pull/5296>`__)
+* `@stsewd <http://github.com/stsewd>`__: Downgrade pytest-django (`#5294 <https://github.com/rtfd/readthedocs.org/pull/5294>`__)
+* `@rexzing <http://github.com/rexzing>`__: Docs reformatting with :guilabel: (`#5161 <https://github.com/rtfd/readthedocs.org/pull/5161>`__)
+
 Version 3.2.2
 -------------
 
@@ -986,14 +1001,12 @@ Version 2.3.1
 * `@humitos <http://github.com/humitos>`__: Upgrade all packages using `pur` tool (`#2916 <https://github.com/rtfd/readthedocs.org/pull/2916>`__)
 * `@rixx <http://github.com/rixx>`__: Fix page redirect preview (`#2711 <https://github.com/rtfd/readthedocs.org/pull/2711>`__)
 
-.. _version-2.3.0:
-
 Version 2.3.0
 -------------
 
 .. warning::
     Version 2.3.0 includes a security fix for project translations. See
-    :ref:`security-2.3.0` for more information
+    :ref:`security:Release 2.3.0` for more information
 
 * `@stsewd <http://github.com/stsewd>`__: Fix tox env for coverage (`#3772 <https://github.com/rtfd/readthedocs.org/pull/3772>`__)
 * `@humitos <http://github.com/humitos>`__: Try to fix end of file (`#3761 <https://github.com/rtfd/readthedocs.org/pull/3761>`__)

common

@@ -0,0 +1,19 @@
+Sitemaps
+========
+
+Sitemaps_ allows us to inform search engines about URLs that are available for crawling
+and communicate them additional information about each URL of the project:
+
+* when it was last updated,
+* how often it changes,
+* how important it is in relation to other URLs in the site, and
+* what translations are available for a page.
+
+Read the Docs automatically generates a sitemap for each project that hosts
+to improve results when performing a search on these search engines.
+This allow us to prioritize results based on the version number, for example
+to show ``latest`` as the top result followed by ``stable`` and then all the project's
+versions sorted following semver_.
+
+.. _semver: https://semver.org/
+.. _Sitemaps: https://www.sitemaps.org/

docs/features/sitemaps.rst

@@ -87,7 +87,7 @@ Security issue archive
 Version 3.2.0
 ~~~~~~~~~~~~~
 
-:ref:`version-3.2.0` resolved an issue where a specially crafted request
+:ref:`changelog:Version 3.2.0` resolved an issue where a specially crafted request
 could result in a DNS query to an arbitrary domain.
 
 This issue was found by `Cyber Smart Defence <https://www.cybersmartdefence.com/>`_
@@ -98,7 +98,7 @@ of Read the Docs.
 Release 2.3.0
 ~~~~~~~~~~~~~
 
-:ref:`version-2.3.0` resolves a security issue with translations on our community
+:ref:`changelog:Version 2.3.0` resolves a security issue with translations on our community
 hosting site that allowed users to modify the hosted path of a target project by
 adding it as a translation project of their own project. A check was added to
 ensure project ownership before adding the project as a translation.

docs/security.rst

@@ -67,14 +67,6 @@ Default: :djangosetting:`DOCUMENT_PYQUERY_PATH`
 The Pyquery path to an HTML element that is the root of your document. 
 This is used for making sure we are only searching the main content of a document.
 
-USE_PIP_INSTALL
----------------
-
-Default: :djangosetting:`USE_PIP_INSTALL`
-
-Whether to use `pip install .` or `python setup.py install` when installing packages into the Virtualenv. Default is to use `python setup.py install`.
-
-
 PUBLIC_DOMAIN
 -------------
 
@@ -177,4 +169,4 @@ This setting is used for automatically indexing objects to elasticsearch.
 project and build documentations without having elasticsearch.
 
 
-.. _elasticsearch-dsl-py.connections.configure: https://elasticsearch-dsl.readthedocs.io/en/stable/configuration.html#multiple-clusters
+.. _elasticsearch-dsl-py.connections.configure: https://elasticsearch-dsl.readthedocs.io/en/stable/configuration.html#multiple-clusters

docs/settings.rst

@@ -22,15 +22,19 @@ argument::
 
     tox "'--including-search'"
 
+.. warning::
+
+   Running tests for search needs an Elasticsearch :ref:`instance running locally <development/search:Installing and running Elasticsearch>`.
+
 To target a specific environment::
 
-    tox -e py27
+    tox -e py36
 
 The ``tox`` configuration has the following environments configured. You can
 target a single environment to limit the test suite::
 
-    py27
-        Run our test suite using Python 2.7
+    py36
+        Run our test suite using Python 3.6
 
     lint
         Run code linting using `Prospector`_. This currently runs `pylint`_,

docs/tests.rst

@@ -702,6 +702,14 @@ p.build-missing { font-size: .8em; color: #9d9a55; margin: 0 0 3px; }
 #footer label { color: #BCC1C3; font-weight: normal; }
 #footer input[type="text"], #footer input[type="email"] { padding: 4px; font-size: 12px; line-height: 16px; margin-bottom: 5px }
 
+/* Warning Icon for Build List triggered */
+.module-item.col-span a span.icon-warning:before {
+  font-family: FontAwesome;
+  font-size: .9em;
+  padding-right: .3em;
+  font-weight: normal;
+  content: "\f071";
+}
 
 /* utils */
 

media/css/core.css

@@ -2,6 +2,7 @@
 
 """Models for the builds app."""
 
+import datetime
 import logging
 import os.path
 import re
@@ -31,6 +32,7 @@
     BRANCH,
     BUILD_STATE,
     BUILD_STATE_FINISHED,
+    BUILD_STATE_TRIGGERED,
     BUILD_TYPES,
     LATEST,
     NON_REPOSITORY_VERSIONS,
@@ -629,6 +631,12 @@ def finished(self):
         """Return if build has a finished state."""
         return self.state == BUILD_STATE_FINISHED
 
+    @property
+    def is_stale(self):
+        """Return if build state is triggered & date more than 5m ago."""
+        mins_ago = timezone.now() - datetime.timedelta(minutes=5)
+        return self.state == BUILD_STATE_TRIGGERED and self.date < mins_ago
+
 
 class BuildCommandResultMixin:
 

readthedocs/builds/models.py

@@ -26,6 +26,7 @@
 
 from django.db import models
 from django.utils.encoding import force_text
+from slugify import slugify as unicode_slugify
 
 
 def get_fields_with_model(cls):
@@ -53,13 +54,15 @@ def get_fields_with_model(cls):
 
 class VersionSlugField(models.CharField):
 
-    """Inspired by ``django_extensions.db.fields.AutoSlugField``."""
+    """
+    Inspired by ``django_extensions.db.fields.AutoSlugField``.
 
-    invalid_chars_re = re.compile('[^-._a-z0-9]')
-    leading_punctuation_re = re.compile('^[-._]+')
-    placeholder = '-'
-    fallback_slug = 'unknown'
+    Uses ``unicode-slugify`` to generate the slug.
+    """
+
+    ok_chars = '-._'  # dash, dot, underscore
     test_pattern = re.compile('^{pattern}$'.format(pattern=VERSION_SLUG_REGEX))
+    fallback_slug = 'unknown'
 
     def __init__(self, *args, **kwargs):
         kwargs.setdefault('db_index', True)
@@ -78,13 +81,42 @@ def get_queryset(self, model_cls, slug_field):
                 return model._default_manager.all()
         return model_cls._default_manager.all()
 
+    def _normalize(self, content):
+        """
+        Normalize some invalid characters (/, %, !, ?) to become a dash (``-``).
+
+        .. note::
+
+            We replace these characters to a dash to keep compatibility with the
+            old behavior and also because it makes this more readable.
+
+        For example, ``release/1.0`` will become ``release-1.0``.
+        """
+        return re.sub('[/%!?]', '-', content)
+
     def slugify(self, content):
+        """
+        Make ``content`` a valid slug.
+
+        It uses ``unicode-slugify`` behind the scenes which works properly with
+        Unicode characters.
+        """
         if not content:
             return ''
 
-        slugified = content.lower()
-        slugified = self.invalid_chars_re.sub(self.placeholder, slugified)
-        slugified = self.leading_punctuation_re.sub('', slugified)
+        normalized = self._normalize(content)
+        slugified = unicode_slugify(
+            normalized,
+            only_ascii=True,
+            spaces=False,
+            lower=True,
+            ok=self.ok_chars,
+            space_replacement='-',
+        )
+
+        # Remove first character wile it's an invalid character for the
+        # beginning of the slug
+        slugified = slugified.lstrip(self.ok_chars)
 
         if not slugified:
             return self.fallback_slug

readthedocs/builds/version_slug.py

@@ -489,15 +489,14 @@ def validate_conda(self):
             raw_conda = self.raw_config['conda']
             with self.catch_validation_error('conda'):
                 validate_dict(raw_conda)
-            conda_environment = None
-            if 'file' in raw_conda:
-                with self.catch_validation_error('conda.file'):
-                    conda_environment = validate_file(
-                        raw_conda['file'],
-                        self.base_path,
-                    )
-            conda['environment'] = conda_environment
-
+            with self.catch_validation_error('conda.file'):
+                if 'file' not in raw_conda:
+                    raise ValidationError('file', VALUE_NOT_FOUND)
+                conda_environment = validate_file(
+                    raw_conda['file'],
+                    self.base_path,
+                )
+                conda['environment'] = conda_environment
             return conda
         return None
 

readthedocs/config/config.py

@@ -557,29 +557,33 @@ def test_it_priorities_image_from_env_config(self, tmpdir, image):
         assert build.build.image == image
 
 
-def test_use_conda_default_false():
+def test_use_conda_default_none():
     build = get_build_config({})
     build.validate()
     assert build.conda is None
 
 
-def test_use_conda_respects_config():
+def test_validates_conda_file(tmpdir):
+    apply_fs(tmpdir, {'environment.yml': ''})
     build = get_build_config(
-        {'conda': {}},
+        {'conda': {'file': 'environment.yml'}},
+        source_file=str(tmpdir.join('readthedocs.yml')),
     )
     build.validate()
     assert isinstance(build.conda, Conda)
+    assert build.conda.environment == str(tmpdir.join('environment.yml'))
 
 
-def test_validates_conda_file(tmpdir):
+def test_file_is_required_when_using_conda(tmpdir):
     apply_fs(tmpdir, {'environment.yml': ''})
     build = get_build_config(
-        {'conda': {'file': 'environment.yml'}},
+        {'conda': {'foo': 'environment.yml'}},
         source_file=str(tmpdir.join('readthedocs.yml')),
     )
-    build.validate()
-    assert isinstance(build.conda, Conda)
-    assert build.conda.environment == str(tmpdir.join('environment.yml'))
+    with raises(InvalidConfig) as excinfo:
+        build.validate()
+    assert excinfo.value.key == 'conda.file'
+    assert excinfo.value.code == VALUE_NOT_FOUND
 
 
 def test_requirements_file_empty():

readthedocs/config/tests/test_config.py

@@ -23,7 +23,7 @@ def add_arguments(self, parser):
         parser.add_argument(
             '--days',
             dest='days',
-            type='int',
+            type=int,
             default=365,
             help='Find builds older than DAYS days, default: 365',
         )

readthedocs/core/management/commands/clean_builds.py

@@ -1,5 +1,3 @@
-# -*- coding: utf-8 -*-
-
 """URL configurations for subdomains."""
 from functools import reduce
 from operator import add
@@ -15,14 +13,16 @@
     redirect_project_slug,
     robots_txt,
     serve_docs,
+    sitemap_xml,
 )
 
 
 handler500 = server_error_500
 handler404 = server_error_404
 
 subdomain_urls = [
-    url(r'robots.txt$', robots_txt, name='robots_txt'),
+    url(r'robots\.txt$', robots_txt, name='robots_txt'),
+    url(r'sitemap\.xml$', sitemap_xml, name='sitemap_xml'),
     url(
         r'^(?:|projects/(?P<subproject_slug>{project_slug})/)'
         r'page/(?P<filename>.*)$'.format(**pattern_opts),