Merge branch 'master' into use-setting-for-footer-api

Author: stsewd

CHANGELOG.rst

@@ -1,3 +1,23 @@
+Version 3.7.4
+-------------
+
+:Date: September 05, 2019
+
+* `@ericholscher <http://github.com/ericholscher>`__: Remove paid support callout (`#6140 <https://github.com/readthedocs/readthedocs.org/pull/6140>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Fix IntegrationAdmin with raw_id_fields for Projects (`#6136 <https://github.com/readthedocs/readthedocs.org/pull/6136>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Fix link to html_extra_path (`#6135 <https://github.com/readthedocs/readthedocs.org/pull/6135>`__)
+* `@stsewd <http://github.com/stsewd>`__: Move out authorization from FooterHTML view (`#6133 <https://github.com/readthedocs/readthedocs.org/pull/6133>`__)
+* `@agjohnson <http://github.com/agjohnson>`__: Add setting for always cleaning the build post-build (`#6132 <https://github.com/readthedocs/readthedocs.org/pull/6132>`__)
+* `@pyup-bot <http://github.com/pyup-bot>`__: pyup:  Scheduled weekly dependency update for week 35 (`#6129 <https://github.com/readthedocs/readthedocs.org/pull/6129>`__)
+* `@stsewd <http://github.com/stsewd>`__:  Refactor footer_html view to class (`#6125 <https://github.com/readthedocs/readthedocs.org/pull/6125>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Use raw_id_fields in the TokenAdmin (`#6116 <https://github.com/readthedocs/readthedocs.org/pull/6116>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Fixed footer ads supported on all themes (`#6115 <https://github.com/readthedocs/readthedocs.org/pull/6115>`__)
+* `@stsewd <http://github.com/stsewd>`__: Don't log BuildEnvironmentWarning as error (`#6112 <https://github.com/readthedocs/readthedocs.org/pull/6112>`__)
+* `@pllim <http://github.com/pllim>`__: Use the force when fetching with Git (`#6109 <https://github.com/readthedocs/readthedocs.org/pull/6109>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Record search queries smartly (`#6088 <https://github.com/readthedocs/readthedocs.org/pull/6088>`__)
+* `@stsewd <http://github.com/stsewd>`__: Add move method to automation rule (`#5998 <https://github.com/readthedocs/readthedocs.org/pull/5998>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Index more domain data into elasticsearch (`#5979 <https://github.com/readthedocs/readthedocs.org/pull/5979>`__)
+
 Version 3.7.3
 -------------
 

docs/custom_installs/index.rst

@@ -12,9 +12,8 @@ It has those features and more!
 
 .. warning::
     These documents are maintained by the community, and might not be up to date.
-    Read the Docs developers do not support custom installs of our software
-    in our public issue tracker. For additional support of a custom installation,
-    please see our `paid support plans <https://readthedocs.com/services/#open-source-support>`_.
+    Read the Docs developers do not support custom installs of our software,
+    as mentioned in our :doc:`/open-source-philosophy`.
 
 .. toctree::
     :maxdepth: 2

docs/faq.rst

@@ -257,7 +257,7 @@ To accomplish this, you can add a ``robots.txt`` file to your documentation's ro
 
 
 Minimal example of ``robots.txt``
-+++++++++++++++++++++++++++++++++
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
 
@@ -270,22 +270,28 @@ Minimal example of ``robots.txt``
    See `Google's docs`_ for its full syntax.
 
 This file has to be served as is under ``/robots.txt``.
-Depending if you are using Sphinx or MkDocs, you will need a different configuration for this.
 
+Setup
+~~~~~
 
-Sphinx
-~~~~~~
+The ``robots.txt`` file will be served from the **default version** of your Project.
+This is because the ``robots.txt`` file is served at the top-level of your domain,
+so we must choose a version to find the file in.
+The **default version** is the best place to look for it.
+
+Sphinx and Mkdocs both have different ways of outputting static files in the build:
 
-Sphinx uses `html_extra`_ option to add static files to the output.
-You need to create a ``robots.txt`` file and put it under the path defined in ``html_extra``.
+Sphinx
+++++++
 
+Sphinx uses `html_extra_path`_ option to add static files to the output.
+You need to create a ``robots.txt`` file and put it under the path defined in ``html_extra_path``.
 
 MkDocs
-~~~~~~
+++++++
 
 MkDocs needs the ``robots.txt`` to be at the directory defined at `docs_dir`_ config.
 
-
 .. _Google's docs: https://support.google.com/webmasters/answer/6062608
-.. _html_extra: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path
+.. _html_extra_path: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_extra_path
 .. _docs_dir: https://www.mkdocs.org/user-guide/configuration/#docs_dir

readthedocs/core/static-src/core/js/doc-embed/rtd-data.js

@@ -22,10 +22,6 @@ var configMethods = {
         return constants.ALABASTER_LIKE_THEMES.indexOf(this.get_theme_name()) > -1;
     },
 
-    theme_supports_paid_promo: function () {
-        return this.is_rtd_like_theme() || this.is_alabaster_like_theme();
-    },
-
     is_sphinx_builder: function () {
         return (!('builder' in this) || this.builder !== 'mkdocs');
     },

readthedocs/core/static-src/core/js/doc-embed/sponsorship.js

@@ -320,10 +320,6 @@ function init() {
     request_data.project = rtd.project;
     request_data.theme = rtd.get_theme_name();
 
-    // These will get community only ads temporarily
-    // After the fixed footer rollout is complete, this can be removed
-    request_data.community_only = (rtd.theme_supports_paid_promo() || explicit_placement) ? 0 : 1;
-
     if (typeof URL !== 'undefined' && typeof URLSearchParams !== 'undefined') {
         // Force a specific promo to be displayed
         params = new URL(window.location).searchParams;

readthedocs/core/static/core/js/readthedocs-doc-embed.js

@@ -1,6 +1,7 @@
 import itertools
 import logging
 
+from django.utils import timezone
 from rest_framework import generics, serializers
 from rest_framework.exceptions import ValidationError
 from rest_framework.pagination import PageNumberPagination
@@ -160,15 +161,19 @@ def list(self, request, *args, **kwargs):
 
         project_slug = self.request.query_params.get('project', None)
         version_slug = self.request.query_params.get('version', None)
-        query = self.request.query_params.get('q', '')
         total_results = response.data.get('count', 0)
+        time = timezone.now()
+
+        query = self.request.query_params.get('q', '')
+        query = query.lower().strip()
 
         # record the search query with a celery task
         tasks.record_search_query.delay(
             project_slug,
             version_slug,
             query,
             total_results,
+            time.isoformat(),
         )
 
         return response

readthedocs/projects/static/projects/js/tools.js

@@ -1,5 +1,6 @@
 import logging
 
+from dateutil.parser import parse
 from django.apps import apps
 from django.utils import timezone
 from django_elasticsearch_dsl.registries import registry
@@ -142,20 +143,46 @@ def delete_old_search_queries_from_db():
 
 
 @app.task(queue='web')
-def record_search_query(project_slug, version_slug, query, total_results):
-    """Record search query in database."""
-    if not project_slug or not version_slug or not query or not total_results:
+def record_search_query(project_slug, version_slug, query, total_results, time_string):
+    """Record/update search query in database."""
+    if not project_slug or not version_slug or not query:
         log.debug(
             'Not recording the search query. Passed arguments: '
-            'project_slug: %s, version_slug: %s, query: %s, total_results: %s' % (
-                project_slug, version_slug, query, total_results
+            'project_slug: %s, version_slug: %s, query: %s, total_results: %s, time: %s' % (
+                project_slug, version_slug, query, total_results, time_string
             )
         )
         return
 
-    project_qs = Project.objects.filter(slug=project_slug)
+    time = parse(time_string)
+    before_10_sec = time - timezone.timedelta(seconds=10)
+    partial_query_qs = SearchQuery.objects.filter(
+        project__slug=project_slug,
+        version__slug=version_slug,
+        created__gte=before_10_sec,
+    ).order_by('-created')
+
+    # check if partial query exists,
+    # if yes, then just update the object.
+    for partial_query in partial_query_qs.iterator():
+        if query.startswith(partial_query.query):
+            partial_query.created = time
+            partial_query.query = query
+            partial_query.save()
+            return
+
+    # don't record query with zero results.
+    if not total_results:
+        log.debug(
+            'Not recording search query because of zero results. Passed arguments: '
+            'project_slug: %s, version_slug: %s, query: %s, total_results: %s, time: %s' % (
+                project_slug, version_slug, query, total_results, time
+            )
+        )
+        return
 
-    if not project_qs.exists():
+    project = Project.objects.filter(slug=project_slug).first()
+    if not project:
         log.debug(
             'Not recording the search query because project does not exist. '
             'project_slug: %s' % (
@@ -164,15 +191,24 @@ def record_search_query(project_slug, version_slug, query, total_results):
         )
         return
 
-    project = project_qs.first()
     version_qs = Version.objects.filter(project=project, slug=version_slug)
 
     if not version_qs.exists():
+        log.debug(
+            'Not recording the search query because version does not exist. '
+            'project_slug: %s, version_slug: %s' % (
+                project_slug, version_slug
+            )
+        )
         return
 
     version = version_qs.first()
-    SearchQuery.objects.create(
+
+    # make a new SearchQuery object.
+    obj = SearchQuery.objects.create(
         project=project,
         version=version,
         query=query,
     )
+    obj.created = time
+    obj.save()

readthedocs/search/api.py

@@ -1,5 +1,6 @@
 """Tests for search tasks."""
 
+import mock
 import pytest
 
 from django.urls import reverse
@@ -43,6 +44,48 @@ def test_search_query_recorded_when_results_not_zero(self, api_client):
             SearchQuery.objects.all().count() == 1
         ), 'there should be 1 obj since a search is made which returns one result.'
 
+    def test_partial_queries_are_not_recorded(self, api_client):
+        """Test if partial queries are not recorded."""
+
+        assert (
+            SearchQuery.objects.all().count() == 0
+        ), 'no SearchQuery should be present if there is no search made.'
+
+        time = timezone.now()
+        search_params = { 'q': 'stack', 'project': 'docs', 'version': 'latest' }
+
+        with mock.patch('django.utils.timezone.now') as test_time:
+            test_time.return_value = time
+            resp = api_client.get(self.url, search_params)
+            assert resp.status_code, 200
+
+        assert (
+            SearchQuery.objects.all().count() == 1
+        ), 'one SearchQuery should be present'
+
+        # update the time and the search query and make another search request
+        time = time + timezone.timedelta(seconds=2)
+        search_params['q'] = 'stack over'
+        with mock.patch('django.utils.timezone.now') as test_time:
+            test_time.return_value = time
+            resp = api_client.get(self.url, search_params)
+            assert resp.status_code, 200
+
+        # update the time and the search query and make another search request
+        time = time + timezone.timedelta(seconds=2)
+        search_params['q'] = 'stack overflow'
+        with mock.patch('django.utils.timezone.now') as test_time:
+            test_time.return_value = time
+            resp = api_client.get(self.url, search_params)
+            assert resp.status_code, 200
+
+        assert (
+            SearchQuery.objects.all().count() == 1
+        ), 'one SearchQuery should be present'
+        assert (
+            SearchQuery.objects.all().first().query == 'stack overflow'
+        ), 'one SearchQuery should be there because partial queries gets updated'
+
     def test_search_query_not_recorded_when_results_are_zero(self, api_client):
         """Test that search queries are not recorded when they have zero results."""
 

readthedocs/search/tasks.py

@@ -1,10 +1,11 @@
-# -*- coding: utf-8 -*-
-
 """Base classes for VCS backends."""
 import logging
 import os
 import shutil
 
+from readthedocs.doc_builder.exceptions import BuildEnvironmentWarning
+from readthedocs.projects.exceptions import RepositoryError
+
 
 log = logging.getLogger(__name__)
 
@@ -102,7 +103,13 @@ def run(self, *cmd, **kwargs):
             'shell': False,
         })
 
-        build_cmd = self.environment.run(*cmd, **kwargs)
+        try:
+            build_cmd = self.environment.run(*cmd, **kwargs)
+        except BuildEnvironmentWarning as e:
+            # Re raise as RepositoryError,
+            # so isn't logged as ERROR.
+            raise RepositoryError(str(e))
+
         # Return a tuple to keep compatibility
         return (build_cmd.exit_code, build_cmd.output, build_cmd.error)
 

readthedocs/search/tests/test_search_tasks.py

@@ -1,6 +1,6 @@
 [metadata]
 name = readthedocs
-version = 3.7.3
+version = 3.7.4
 license = MIT
 description = Read the Docs builds and hosts documentation
 author = Read the Docs, Inc