Merge branch 'master' into refactor-views-search-analytics

Author: stsewd

CHANGELOG.rst

@@ -1,3 +1,27 @@
+Version 3.7.5
+-------------
+
+:Date: September 26, 2019
+
+* `@davidfischer <http://github.com/davidfischer>`__: Remove if storage blocks (`#6191 <https://github.com/readthedocs/readthedocs.org/pull/6191>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Update security docs (`#6179 <https://github.com/readthedocs/readthedocs.org/pull/6179>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Add the private spamfighting module to INSTALLED_APPS (`#6177 <https://github.com/readthedocs/readthedocs.org/pull/6177>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Document connected account permissions (`#6172 <https://github.com/readthedocs/readthedocs.org/pull/6172>`__)
+* `@stsewd <http://github.com/stsewd>`__: Require login for old redirect (`#6170 <https://github.com/readthedocs/readthedocs.org/pull/6170>`__)
+* `@humitos <http://github.com/humitos>`__: Remove old and unused code (`#6167 <https://github.com/readthedocs/readthedocs.org/pull/6167>`__)
+* `@stsewd <http://github.com/stsewd>`__: Clean up views (`#6166 <https://github.com/readthedocs/readthedocs.org/pull/6166>`__)
+* `@stsewd <http://github.com/stsewd>`__: Update docs for sharing (`#6164 <https://github.com/readthedocs/readthedocs.org/pull/6164>`__)
+* `@pyup-bot <http://github.com/pyup-bot>`__: pyup:  Scheduled weekly dependency update for week 36 (`#6158 <https://github.com/readthedocs/readthedocs.org/pull/6158>`__)
+* `@saadmk11 <http://github.com/saadmk11>`__: Remove PR Builder Project Idea from RTD GSoC Docs (`#6147 <https://github.com/readthedocs/readthedocs.org/pull/6147>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Serialize time in search queries properly (`#6142 <https://github.com/readthedocs/readthedocs.org/pull/6142>`__)
+* `@humitos <http://github.com/humitos>`__: Allow to extend DomainCreate view (`#6139 <https://github.com/readthedocs/readthedocs.org/pull/6139>`__)
+* `@saadmk11 <http://github.com/saadmk11>`__: Integration Re-sync Bug Fix (`#6124 <https://github.com/readthedocs/readthedocs.org/pull/6124>`__)
+* `@stsewd <http://github.com/stsewd>`__: Don't log BuildEnvironmentWarning as error (`#6112 <https://github.com/readthedocs/readthedocs.org/pull/6112>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Add Search Guide (`#6101 <https://github.com/readthedocs/readthedocs.org/pull/6101>`__)
+* `@saadmk11 <http://github.com/saadmk11>`__: Add PR Builder guide to docs (`#6093 <https://github.com/readthedocs/readthedocs.org/pull/6093>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Record search queries smartly (`#6088 <https://github.com/readthedocs/readthedocs.org/pull/6088>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Remove 'highlight' URL param from search results (`#6087 <https://github.com/readthedocs/readthedocs.org/pull/6087>`__)
+
 Version 3.7.4
 -------------
 

docs/connected-accounts.rst

@@ -1,5 +1,5 @@
 Connecting Your Account
------------------------
+=======================
 
 If you are going to import repositories from GitHub, Bitbucket, or GitLab,
 you should connect your Read the Docs account to your repository host first.
@@ -18,3 +18,69 @@ and select `Connected Services <https://readthedocs.org/accounts/social/connecti
 From here, you'll be able to connect to your GitHub, Bitbucket or GitLab
 account. This process will ask you to authorize a connection to Read the Docs,
 that allows us to read information about and clone your repositories.
+
+
+Permissions for connected accounts
+----------------------------------
+
+Read the Docs does not generally ask for write permission to your repositories' code
+(with one exception detailed below)
+and since we only connect to public repositories we don't need special permissions to read them.
+However, we do need permissions for authorizing your account
+so that you can login to Read the Docs with your connected account credentials
+and to setup :doc:`webhooks`
+which allow us to build your documentation on every change to your repository.
+
+
+GitHub
+~~~~~~
+
+Read the Docs requests the following permissions (more precisely, `OAuth scopes`_)
+when connecting your Read the Docs account to GitHub.
+
+.. _OAuth scopes: https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/
+
+Read access to your email address (``user:email``)
+    We ask for this so you can create a Read the Docs account and login with your GitHub credentials.
+
+Administering webhooks (``admin:repo_hook``)
+    We ask for this so we can create webhooks on your repositories when you import them into Read the Docs.
+    This allows us to build the docs when you push new commits.
+
+Read access to your organizations (``read:org``)
+    We ask for this so we know which organizations you have access to.
+    This allows you to filter repositories by organization when importing repositories.
+
+Repository status (``repo:status``)
+    Repository statuses allow Read the Docs to report the status
+    (eg. passed, failed, pending) of pull requests to GitHub.
+    This is used for a feature currently in beta testing
+    that builds documentation on each pull request similar to a continuous integration service.
+
+.. note::
+
+    :doc:`Read the Docs for Business </commercial/index>`
+    asks for one additional permission (``repo``) to allow access to private repositories
+    and to allow us to setup SSH keys to clone your private repositories.
+    Unfortunately, this is the permission for read/write control of the repository
+    but there isn't a more granular permission
+    that only allows setting up SSH keys for read access.
+
+
+Bitbucket
+~~~~~~~~~
+
+For similar reasons to those above for GitHub, we request permissions for:
+
+* Reading your account information including your email address
+* Read access to your team memberships
+* Read access to your repositories
+* Read and write access to webhooks
+
+GitLab
+~~~~~~
+
+Like the others, we request permissions for:
+
+* Reading your account information (``read_user``)
+* API access (``api``) which is needed to create webhooks in GitLab

docs/faq.rst

@@ -81,30 +81,14 @@ I get import errors on libraries that depend on C modules
 ---------------------------------------------------------
 
 .. note::
-    Another use case for this is when you have a module with a C extension.
 
-This happens because our build system doesn't have the dependencies for building your project. This happens with things like ``libevent``, ``mysql``, and other python packages that depend on C libraries. We can't support installing random C binaries on our system, so there is another way to fix these imports.
+   Another use case for this is when you have a module with a C extension.
 
-With Sphinx you can use the built-in `autodoc_mock_imports`_ for mocking. Alternatively you can use the mock library by putting the following snippet in your ``conf.py``::
-
-    import sys
-    from unittest.mock import MagicMock
-
-    class Mock(MagicMock):
-        @classmethod
-        def __getattr__(cls, name):
-            return MagicMock()
-
-    MOCK_MODULES = ['pygtk', 'gtk', 'gobject', 'argparse', 'numpy', 'pandas']
-    sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
-
-You need to replace ``MOCK_MODULES`` with the modules that you want to mock out.
-
-.. Tip:: The library ``unittest.mock`` was introduced on python 3.3. On earlier versions install the ``mock`` library
-    from PyPI with (ie ``pip install mock``) and replace the above import::
-
-        from mock import Mock as MagicMock
+This happens because our build system doesn't have the dependencies for building your project.
+This happens with things like ``libevent``, ``mysql``, and other python packages that depend on C libraries.
+We can't support installing random C binaries on our system, so there is another way to fix these imports.
 
+With Sphinx you can use the built-in `autodoc_mock_imports`_ for mocking.
 If such libraries are installed via ``setup.py``, you also will need to remove all the C-dependent libraries from your ``install_requires`` in the RTD environment.
 
 .. _autodoc_mock_imports: http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_mock_imports

package.json

@@ -1,6 +1,6 @@
 {
   "name": "readthedocs",
-  "version": "0.0.1",
+  "version": "3.7.5",
   "description": "Read the Docs build dependencies",
   "author": "Anthony Johnson <[email protected]>",
   "scripts": {

readthedocs/builds/storage.py

@@ -1,6 +1,7 @@
 import logging
 from pathlib import Path
 
+from django.conf import settings
 from django.core.exceptions import SuspiciousFileOperation
 from django.core.files.storage import FileSystemStorage
 from storages.utils import safe_join, get_available_overwrite_name
@@ -107,7 +108,19 @@ def walk(self, top):
 
 class BuildMediaFileSystemStorage(BuildMediaStorageMixin, FileSystemStorage):
 
-    """Storage subclass that writes build artifacts under MEDIA_ROOT."""
+    """Storage subclass that writes build artifacts in PRODUCTION_MEDIA_ARTIFACTS or MEDIA_ROOT."""
+
+    def __init__(self, **kwargs):
+        location = kwargs.pop('location', None)
+
+        if not location:
+            # Mirrors the logic of getting the production media path
+            if settings.DEFAULT_PRIVACY_LEVEL == 'public' or settings.DEBUG:
+                location = settings.MEDIA_ROOT
+            else:
+                location = settings.PRODUCTION_MEDIA_ARTIFACTS
+
+        super().__init__(location)
 
     def get_available_name(self, name, max_length=None):
         """

readthedocs/builds/views.py

@@ -55,8 +55,7 @@ def post(self, request, project_slug):
 
         version_slug = request.POST.get('version_slug')
         version = get_object_or_404(
-            Version.internal.all(),
-            project=project,
+            self._get_versions(project),
             slug=version_slug,
         )
 
@@ -79,6 +78,12 @@ def post(self, request, project_slug):
             reverse('builds_detail', args=[project.slug, build.pk]),
         )
 
+    def _get_versions(self, project):
+        return Version.internal.public(
+            user=self.request.user,
+            project=project,
+        )
+
 
 class BuildList(BuildBase, BuildTriggerMixin, ListView):
 
@@ -91,10 +96,7 @@ def get_context_data(self, **kwargs):
 
         context['project'] = self.project
         context['active_builds'] = active_builds
-        context['versions'] = Version.internal.public(
-            user=self.request.user,
-            project=self.project,
-        )
+        context['versions'] = self._get_versions(self.project)
         context['build_qs'] = self.get_queryset()
 
         return context

readthedocs/core/static/core/js/readthedocs-doc-embed.js

@@ -783,15 +783,12 @@ def has_media(self, type_, version_slug=LATEST, version_type=None):
         if os.path.exists(path):
             return True
 
-        if settings.RTD_BUILD_MEDIA_STORAGE:
-            storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
-            storage_path = self.get_storage_path(
-                type_=type_, version_slug=version_slug,
-                version_type=version_type
-            )
-            return storage.exists(storage_path)
-
-        return False
+        storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
+        storage_path = self.get_storage_path(
+            type_=type_, version_slug=version_slug,
+            version_type=version_type
+        )
+        return storage.exists(storage_path)
 
     def has_pdf(self, version_slug=LATEST, version_type=None):
         return self.has_media(
@@ -1282,33 +1279,27 @@ def get_processed_json(self):
         https://github.com/rtfd/readthedocs.org/issues/5368
         """
         file_path = None
+        storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
 
-        if settings.RTD_BUILD_MEDIA_STORAGE:
-            storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
-
-            fjson_paths = []
-            basename = os.path.splitext(self.path)[0]
-            fjson_paths.append(basename + '.fjson')
-            if basename.endswith('/index'):
-                new_basename = re.sub(r'\/index$', '', basename)
-                fjson_paths.append(new_basename + '.fjson')
+        fjson_paths = []
+        basename = os.path.splitext(self.path)[0]
+        fjson_paths.append(basename + '.fjson')
+        if basename.endswith('/index'):
+            new_basename = re.sub(r'\/index$', '', basename)
+            fjson_paths.append(new_basename + '.fjson')
 
-            storage_path = self.project.get_storage_path(
-                type_='json', version_slug=self.version.slug, include_file=False
-            )
-            try:
-                for fjson_path in fjson_paths:
-                    file_path = storage.join(storage_path, fjson_path)
-                    if storage.exists(file_path):
-                        return process_file(file_path)
-            except Exception:
-                log.warning(
-                    'Unhandled exception during search processing file: %s',
-                    file_path,
-                )
-        else:
+        storage_path = self.project.get_storage_path(
+            type_='json', version_slug=self.version.slug, include_file=False
+        )
+        try:
+            for fjson_path in fjson_paths:
+                file_path = storage.join(storage_path, fjson_path)
+                if storage.exists(file_path):
+                    return process_file(file_path)
+        except Exception:
             log.warning(
-                'Skipping HTMLFile processing because of no storage backend'
+                'Unhandled exception during search processing file: %s',
+                file_path,
             )
 
         return {

readthedocs/projects/models.py

@@ -777,6 +777,7 @@ def store_build_artifacts(
         :param epub: whether to save ePub output
         """
         if not settings.RTD_BUILD_MEDIA_STORAGE:
+            # Note: this check can be removed once corporate build servers use storage
             log.warning(
                 LOG_TEMPLATE,
                 {
@@ -1381,10 +1382,6 @@ def _create_intersphinx_data(version, commit, build):
     :param commit: Commit that updated path
     :param build: Build id
     """
-    if not settings.RTD_BUILD_MEDIA_STORAGE:
-        log.warning('RTD_BUILD_MEDIA_STORAGE is missing - Not updating intersphinx data')
-        return
-
     storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
 
     html_storage_path = version.project.get_storage_path(
@@ -1542,11 +1539,6 @@ def _create_imported_files(version, commit, build):
     :returns: paths of changed files
     :rtype: set
     """
-
-    if not settings.RTD_BUILD_MEDIA_STORAGE:
-        log.warning('RTD_BUILD_MEDIA_STORAGE is missing - Not updating imported files')
-        return
-
     storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
 
     changed_files = set()
@@ -1842,10 +1834,6 @@ def remove_build_storage_paths(paths):
 
     :param paths: list of paths in build media storage to delete
     """
-    if not settings.RTD_BUILD_MEDIA_STORAGE:
-        log.warning('RTD_BUILD_MEDIA_STORAGE is missing - Not removing paths from media storage')
-        return
-
     storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)()
     for storage_path in paths:
         log.info('Removing %s from media storage', storage_path)

readthedocs/projects/tasks.py

@@ -26,6 +26,7 @@
     ProjectAdvancedUpdate,
     ProjectAdvertisingUpdate,
     ProjectDashboard,
+    ProjectDelete,
     ProjectUpdate,
     SearchAnalytics,
 )
@@ -69,7 +70,8 @@
         private.project_version_detail, name='project_version_detail',
     ),
     url(
-        r'^(?P<project_slug>[-\w]+)/delete/$', private.project_delete,
+        r'^(?P<project_slug>[-\w]+)/delete/$',
+        ProjectDelete.as_view(),
         name='projects_delete',
     ),
     url(