Merge remote-tracking branch 'origin/master' into intersphinx-modeling

Author: ericholscher

docs/_static/css/sphinx_prompt_css.css

@@ -7,7 +7,8 @@ pre.highlight {
   padding: 12px 12px;
 }
 
-pre.highlight span.prompt1 {
+/* Tweak css for sphinx_prompt #5281 */
+pre.highlight span[class^="prompt"] {
   font-size: 12px;
   line-height: 1.4;
 }

docs/features/sitemaps.rst

@@ -0,0 +1,19 @@
+Sitemaps
+========
+
+Sitemaps_ allows us to inform search engines about URLs that are available for crawling
+and communicate them additional information about each URL of the project:
+
+* when it was last updated,
+* how often it changes,
+* how important it is in relation to other URLs in the site, and
+* what translations are available for a page.
+
+Read the Docs automatically generates a sitemap for each project that hosts
+to improve results when performing a search on these search engines.
+This allow us to prioritize results based on the version number, for example
+to show ``latest`` as the top result followed by ``stable`` and then all the project's
+versions sorted following semver_.
+
+.. _semver: https://semver.org/
+.. _Sitemaps: https://www.sitemaps.org/

docs/gsoc.rst

@@ -1,13 +1,13 @@
 Google Summer of Code
 =====================
 
-.. 
-    .. note:: Thanks for your interest in Read the Docs!
-              We are working hard to update the ideas list now that we are accepted in GSOC.
-              Please give us a little while to work on things,
-              and check back on this page for updates.
+.. note:: Thanks for your interest in Read the Docs!
+          Please follow the instructions in `Getting Started`_,
+          as a good place to start.
+          **Contacting us will not increase your chance of being accepted,
+          but opening Pull Requests with docs and tests will.**
 
-Read the Docs is hoping to be in the Google Summer of Code in 2019.
+Read the Docs is excited to be in the Google Summer of Code in 2019.
 This page will contain all the information for students and anyone else interested in helping.
 
 Skills
@@ -26,18 +26,6 @@ We're happy to help you get up to speed,
 but the more you are able to demonstrate ability in advance,
 the more likely we are to choose your application! 
 
-Mentors
--------
-
-Currently we have a few folks signed up:
-
-* Eric Holscher
-* Manuel Kaufmann
-* Anthony Johnson
-
-.. warning:: Please do not reach out directly to anyone about the Summer of Code.
-             It will **not** increase your chances of being accepted!
-
 Getting Started
 ---------------
 
@@ -53,6 +41,19 @@ Want to get involved?
 
 If you're interested in participating in GSoC as a student, you can apply during the normal process provided by Google. We are currently overwhelmed with interest, so we are not able to respond individually to each person who is interested.
 
+Mentors
+-------
+
+Currently we have a few folks signed up:
+
+* Eric Holscher
+* Manuel Kaufmann
+* Anthony Johnson
+* Safwan Rahman
+
+.. warning:: Please do not reach out directly to anyone about the Summer of Code.
+             It will **not** increase your chances of being accepted!
+
 Project Ideas
 -------------
 

docs/settings.rst

@@ -67,14 +67,6 @@ Default: :djangosetting:`DOCUMENT_PYQUERY_PATH`
 The Pyquery path to an HTML element that is the root of your document. 
 This is used for making sure we are only searching the main content of a document.
 
-USE_PIP_INSTALL
----------------
-
-Default: :djangosetting:`USE_PIP_INSTALL`
-
-Whether to use `pip install .` or `python setup.py install` when installing packages into the Virtualenv. Default is to use `python setup.py install`.
-
-
 PUBLIC_DOMAIN
 -------------
 
@@ -177,4 +169,4 @@ This setting is used for automatically indexing objects to elasticsearch.
 project and build documentations without having elasticsearch.
 
 
-.. _elasticsearch-dsl-py.connections.configure: https://elasticsearch-dsl.readthedocs.io/en/stable/configuration.html#multiple-clusters
+.. _elasticsearch-dsl-py.connections.configure: https://elasticsearch-dsl.readthedocs.io/en/stable/configuration.html#multiple-clusters

docs/tests.rst

@@ -22,15 +22,19 @@ argument::
 
     tox "'--including-search'"
 
+.. warning::
+
+   Running tests for search needs an Elasticsearch :ref:`instance running locally <development/search:Installing and running Elasticsearch>`.
+
 To target a specific environment::
 
-    tox -e py27
+    tox -e py36
 
 The ``tox`` configuration has the following environments configured. You can
 target a single environment to limit the test suite::
 
-    py27
-        Run our test suite using Python 2.7
+    py36
+        Run our test suite using Python 3.6
 
     lint
         Run code linting using `Prospector`_. This currently runs `pylint`_,

media/css/core.css

@@ -702,6 +702,14 @@ p.build-missing { font-size: .8em; color: #9d9a55; margin: 0 0 3px; }
 #footer label { color: #BCC1C3; font-weight: normal; }
 #footer input[type="text"], #footer input[type="email"] { padding: 4px; font-size: 12px; line-height: 16px; margin-bottom: 5px }
 
+/* Warning Icon for Build List triggered */
+.module-item.col-span a span.icon-warning:before {
+  font-family: FontAwesome;
+  font-size: .9em;
+  padding-right: .3em;
+  font-weight: normal;
+  content: "\f071";
+}
 
 /* utils */
 

readthedocs/builds/admin.py

@@ -7,6 +7,7 @@
 
 from readthedocs.builds.models import Build, BuildCommandResult, Version
 from readthedocs.core.utils import trigger_build
+from readthedocs.core.utils.general import wipe_version_via_slugs
 
 
 class BuildCommandResultInline(admin.TabularInline):
@@ -57,7 +58,22 @@ class VersionAdmin(GuardedModelAdmin):
     list_filter = ('type', 'privacy_level', 'active', 'built')
     search_fields = ('slug', 'project__slug')
     raw_id_fields = ('project',)
-    actions = ['build_version']
+    actions = ['wipe_selected_versions', 'build_version']
+
+    def wipe_selected_versions(self, request, queryset):
+        """Wipes the selected versions."""
+        for version in queryset:
+            wipe_version_via_slugs(
+                version_slug=version.slug,
+                project_slug=version.project.slug
+            )
+            self.message_user(
+                request,
+                'Wiped {}.'.format(version.slug),
+                level=messages.SUCCESS
+            )
+
+    wipe_selected_versions.short_description = 'Wipe selected versions'
 
     def build_version(self, request, queryset):
         """Trigger a build for the project version."""

readthedocs/builds/models.py

@@ -2,6 +2,7 @@
 
 """Models for the builds app."""
 
+import datetime
 import logging
 import os.path
 import re
@@ -31,6 +32,7 @@
     BRANCH,
     BUILD_STATE,
     BUILD_STATE_FINISHED,
+    BUILD_STATE_TRIGGERED,
     BUILD_TYPES,
     LATEST,
     NON_REPOSITORY_VERSIONS,
@@ -629,6 +631,12 @@ def finished(self):
         """Return if build has a finished state."""
         return self.state == BUILD_STATE_FINISHED
 
+    @property
+    def is_stale(self):
+        """Return if build state is triggered & date more than 5m ago."""
+        mins_ago = timezone.now() - datetime.timedelta(minutes=5)
+        return self.state == BUILD_STATE_TRIGGERED and self.date < mins_ago
+
 
 class BuildCommandResultMixin:
 

readthedocs/builds/syncers.py

@@ -1,5 +1,3 @@
-# -*- coding: utf-8 -*-
-
 """
 Classes to copy files between build and web servers.
 
@@ -13,12 +11,15 @@
 import shutil
 
 from django.conf import settings
+from django.core.exceptions import SuspiciousFileOperation
+from django.core.files.storage import get_storage_class
 
 from readthedocs.core.utils import safe_makedirs
 from readthedocs.core.utils.extend import SettingsOverrideObject
 
 
 log = logging.getLogger(__name__)
+storage = get_storage_class()()
 
 
 class BaseSyncer:
@@ -42,6 +43,11 @@ def copy(cls, path, target, is_file=False, **kwargs):
                 return
             if os.path.exists(target):
                 os.remove(target)
+
+            # Create containing directory if it doesn't exist
+            directory = os.path.dirname(target)
+            safe_makedirs(directory)
+
             shutil.copy2(path, target)
         else:
             if os.path.exists(target):
@@ -143,6 +149,10 @@ def copy(cls, path, target, host, is_file=False, **kwargs):  # pylint: disable=a
         log.info('Remote Pull %s to %s', path, target)
         if not is_file and not os.path.exists(target):
             safe_makedirs(target)
+        if is_file:
+            # Create containing directory if it doesn't exist
+            directory = os.path.dirname(target)
+            safe_makedirs(directory)
         # Add a slash when copying directories
         sync_cmd = "rsync -e 'ssh -T' -av --delete {user}@{host}:{path} {target}".format(
             host=host,
@@ -159,6 +169,59 @@ def copy(cls, path, target, host, is_file=False, **kwargs):  # pylint: disable=a
             )
 
 
+class SelectiveStorageRemotePuller(RemotePuller):
+
+    """
+    Like RemotePuller but certain files are copied via Django's storage system.
+
+    If a file with extensions specified by ``extensions`` is copied, it will be copied to storage
+    and the original is removed.
+
+    See: https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-DEFAULT_FILE_STORAGE
+    """
+
+    extensions = ('.pdf', '.epub', '.zip')
+
+    @classmethod
+    def get_storage_path(cls, path):
+        """
+        Gets the path to the file within the storage engine.
+
+        For example, if the path was $MEDIA_ROOT/pdfs/latest.pdf
+         the storage_path is 'pdfs/latest.pdf'
+
+        :raises: SuspiciousFileOperation if the path isn't under settings.MEDIA_ROOT
+        """
+        path = os.path.normpath(path)
+        if not path.startswith(settings.MEDIA_ROOT):
+            raise SuspiciousFileOperation
+
+        path = path.replace(settings.MEDIA_ROOT, '').lstrip('/')
+        return path
+
+    @classmethod
+    def copy(cls, path, target, host, is_file=False, **kwargs):  # pylint: disable=arguments-differ
+        RemotePuller.copy(path, target, host, is_file, **kwargs)
+
+        if getattr(storage, 'write_build_media', False):
+            # This is a sanity check for the case where
+            # storage is backed by the local filesystem
+            # In that case, removing the original target file locally
+            # would remove the file from storage as well
+
+            if is_file and os.path.exists(target) and \
+                    any([target.lower().endswith(ext) for ext in cls.extensions]):
+                log.info('Selective Copy %s to media storage', target)
+
+                storage_path = cls.get_storage_path(target)
+
+                if storage.exists(storage_path):
+                    storage.delete(storage_path)
+
+                with open(target, 'rb') as fd:
+                    storage.save(storage_path, fd)
+
+
 class Syncer(SettingsOverrideObject):
     _default_class = LocalSyncer
     _override_setting = 'FILE_SYNCER'

readthedocs/builds/version_slug.py

@@ -26,6 +26,7 @@
 
 from django.db import models
 from django.utils.encoding import force_text
+from slugify import slugify as unicode_slugify
 
 
 def get_fields_with_model(cls):
@@ -53,13 +54,15 @@ def get_fields_with_model(cls):
 
 class VersionSlugField(models.CharField):
 
-    """Inspired by ``django_extensions.db.fields.AutoSlugField``."""
+    """
+    Inspired by ``django_extensions.db.fields.AutoSlugField``.
 
-    invalid_chars_re = re.compile('[^-._a-z0-9]')
-    leading_punctuation_re = re.compile('^[-._]+')
-    placeholder = '-'
-    fallback_slug = 'unknown'
+    Uses ``unicode-slugify`` to generate the slug.
+    """
+
+    ok_chars = '-._'  # dash, dot, underscore
     test_pattern = re.compile('^{pattern}$'.format(pattern=VERSION_SLUG_REGEX))
+    fallback_slug = 'unknown'
 
     def __init__(self, *args, **kwargs):
         kwargs.setdefault('db_index', True)
@@ -78,13 +81,42 @@ def get_queryset(self, model_cls, slug_field):
                 return model._default_manager.all()
         return model_cls._default_manager.all()
 
+    def _normalize(self, content):
+        """
+        Normalize some invalid characters (/, %, !, ?) to become a dash (``-``).
+
+        .. note::
+
+            We replace these characters to a dash to keep compatibility with the
+            old behavior and also because it makes this more readable.
+
+        For example, ``release/1.0`` will become ``release-1.0``.
+        """
+        return re.sub('[/%!?]', '-', content)
+
     def slugify(self, content):
+        """
+        Make ``content`` a valid slug.
+
+        It uses ``unicode-slugify`` behind the scenes which works properly with
+        Unicode characters.
+        """
         if not content:
             return ''
 
-        slugified = content.lower()
-        slugified = self.invalid_chars_re.sub(self.placeholder, slugified)
-        slugified = self.leading_punctuation_re.sub('', slugified)
+        normalized = self._normalize(content)
+        slugified = unicode_slugify(
+            normalized,
+            only_ascii=True,
+            spaces=False,
+            lower=True,
+            ok=self.ok_chars,
+            space_replacement='-',
+        )
+
+        # Remove first character wile it's an invalid character for the
+        # beginning of the slug
+        slugified = slugified.lstrip(self.ok_chars)
 
         if not slugified:
             return self.fallback_slug