Merge branch 'master' into search-reapply

Author: ericholscher

.codecov.yml

@@ -0,0 +1,2 @@
+comment:
+  layout: "diff, files"

.coveragerc

@@ -1,3 +1,20 @@
 [run]
+branch = true
+source = .
 omit =
   */**/migrations/*
+  */**/management/commands/*
+  */**/settings/*
+  */**/settings.py
+  analytics/vendor/*
+  rtd_tests/*
+  */**/tests/**
+  wsgi.py
+
+[report]
+exclude_lines =
+    if self.debug:
+    pragma: no cover
+    raise NotImplementedError
+    if __name__ == .__main__.:
+    if getattr(settings, 'DEBUG'):

.travis.yml

@@ -16,11 +16,15 @@ matrix:
       script: tox
     - python: 3.6
       env: TOXENV=eslint
+    - python: 3.6
+      env: TOXENV=migrations
 cache:
   directories:
     - ~/.cache/pip
     - ~/.nvm/nvm.sh
     - ~/.npm
+before_install:
+  - sudo apt-get install -y git
 install:
   - ./scripts/travis/install_elasticsearch.sh
   - pip install tox-travis
@@ -42,4 +46,6 @@ notifications:
 
 branches:
   only:
-  - master
+  - master 
+  - rel # Community release branch
+  - relcorp # Corporate release branch

CHANGELOG.rst

@@ -1,3 +1,59 @@
+Version 2.8.0
+-------------
+
+:Date: October 30, 2018
+
+Major change is an upgrade to Django 1.11. 
+
+* `@humitos <http://github.com/humitos>`__: Cleanup old code (remove old_div) (`#4817 <https://github.com/rtfd/readthedocs.org/pull/4817>`__)
+* `@humitos <http://github.com/humitos>`__: Remove unnecessary migration (`#4806 <https://github.com/rtfd/readthedocs.org/pull/4806>`__)
+* `@humitos <http://github.com/humitos>`__: Feature flag to make `readthedocs` theme default on MkDocs docs (`#4802 <https://github.com/rtfd/readthedocs.org/pull/4802>`__)
+* `@stsewd <http://github.com/stsewd>`__: Add codecov badge (`#4799 <https://github.com/rtfd/readthedocs.org/pull/4799>`__)
+* `@humitos <http://github.com/humitos>`__: Pin missing dependency for the MkDocs guide compatibility (`#4798 <https://github.com/rtfd/readthedocs.org/pull/4798>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Release 2.7.2 (`#4796 <https://github.com/rtfd/readthedocs.org/pull/4796>`__)
+* `@humitos <http://github.com/humitos>`__: Do not log as error a webhook with an invalid branch name (`#4779 <https://github.com/rtfd/readthedocs.org/pull/4779>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Run travis on release branches (`#4763 <https://github.com/rtfd/readthedocs.org/pull/4763>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Remove Eric & Anthony from ADMINS & MANAGERS settings (`#4762 <https://github.com/rtfd/readthedocs.org/pull/4762>`__)
+* `@stsewd <http://github.com/stsewd>`__: Don't use RequestsContext (`#4759 <https://github.com/rtfd/readthedocs.org/pull/4759>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Django 1.11 upgrade (`#4750 <https://github.com/rtfd/readthedocs.org/pull/4750>`__)
+* `@stsewd <http://github.com/stsewd>`__: Dropdown to select Advanced Settings (`#4710 <https://github.com/rtfd/readthedocs.org/pull/4710>`__)
+* `@stsewd <http://github.com/stsewd>`__: Remove hardcoded constant from config module (`#4704 <https://github.com/rtfd/readthedocs.org/pull/4704>`__)
+* `@stsewd <http://github.com/stsewd>`__: Update tastypie (`#4325 <https://github.com/rtfd/readthedocs.org/pull/4325>`__)
+* `@stsewd <http://github.com/stsewd>`__: Update to Django 1.10 (`#4319 <https://github.com/rtfd/readthedocs.org/pull/4319>`__)
+
+Version 2.7.2
+-------------
+
+:Date: October 23, 2018
+
+* `@humitos <http://github.com/humitos>`__: Validate the slug generated is valid before importing a project (`#4780 <https://github.com/rtfd/readthedocs.org/pull/4780>`__)
+* `@humitos <http://github.com/humitos>`__: Do not log as error a webhook with an invalid branch name (`#4779 <https://github.com/rtfd/readthedocs.org/pull/4779>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Add an index page to our design docs. (`#4775 <https://github.com/rtfd/readthedocs.org/pull/4775>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Remove /embed API endpoint (`#4771 <https://github.com/rtfd/readthedocs.org/pull/4771>`__)
+* `@stsewd <http://github.com/stsewd>`__: Upgrade logs from debug on middleware (`#4769 <https://github.com/rtfd/readthedocs.org/pull/4769>`__)
+* `@humitos <http://github.com/humitos>`__: Link to SSL for Custom Domains fixed (`#4766 <https://github.com/rtfd/readthedocs.org/pull/4766>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Remove Eric & Anthony from ADMINS & MANAGERS settings (`#4762 <https://github.com/rtfd/readthedocs.org/pull/4762>`__)
+* `@humitos <http://github.com/humitos>`__: Do not re-raise the exception if the one that we are checking (`#4761 <https://github.com/rtfd/readthedocs.org/pull/4761>`__)
+* `@humitos <http://github.com/humitos>`__: Do not fail when unlinking an non-existing path (`#4760 <https://github.com/rtfd/readthedocs.org/pull/4760>`__)
+* `@humitos <http://github.com/humitos>`__: Allow to extend the DomainForm from outside (`#4752 <https://github.com/rtfd/readthedocs.org/pull/4752>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Fixes an OSX issue with the test suite (`#4748 <https://github.com/rtfd/readthedocs.org/pull/4748>`__)
+* `@humitos <http://github.com/humitos>`__: Use Docker time limit for max lock age (`#4747 <https://github.com/rtfd/readthedocs.org/pull/4747>`__)
+* `@xyNNN <http://github.com/xyNNN>`__: Fixed link of PagerDuty (`#4744 <https://github.com/rtfd/readthedocs.org/pull/4744>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Make storage syncers extend from a base class (`#4742 <https://github.com/rtfd/readthedocs.org/pull/4742>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Revert "Upgrade theme media to 0.4.2" (`#4735 <https://github.com/rtfd/readthedocs.org/pull/4735>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Upgrade theme media to 0.4.2 (`#4734 <https://github.com/rtfd/readthedocs.org/pull/4734>`__)
+* `@stsewd <http://github.com/stsewd>`__: Extend install option from config file (v2, schema only) (`#4732 <https://github.com/rtfd/readthedocs.org/pull/4732>`__)
+* `@stsewd <http://github.com/stsewd>`__: Remove /cname endpoint (`#4731 <https://github.com/rtfd/readthedocs.org/pull/4731>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Fix get_vcs_repo by moving it to the Mixin (`#4727 <https://github.com/rtfd/readthedocs.org/pull/4727>`__)
+* `@humitos <http://github.com/humitos>`__: Guide explaining how to keep compatibility with mkdocs (`#4726 <https://github.com/rtfd/readthedocs.org/pull/4726>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Release 2.7.1 (`#4725 <https://github.com/rtfd/readthedocs.org/pull/4725>`__)
+* `@dojutsu-user <http://github.com/dojutsu-user>`__: Fix the form for adopting a project (`#4721 <https://github.com/rtfd/readthedocs.org/pull/4721>`__)
+* `@ericholscher <http://github.com/ericholscher>`__: Remove logging verbosity on syncer failure (`#4717 <https://github.com/rtfd/readthedocs.org/pull/4717>`__)
+* `@humitos <http://github.com/humitos>`__: Lint requirement file for py2 (`#4712 <https://github.com/rtfd/readthedocs.org/pull/4712>`__)
+* `@davidfischer <http://github.com/davidfischer>`__: Improve the getting started docs (`#4676 <https://github.com/rtfd/readthedocs.org/pull/4676>`__)
+* `@stsewd <http://github.com/stsewd>`__: Strict validation in configuration file (v2 only) (`#4607 <https://github.com/rtfd/readthedocs.org/pull/4607>`__)
+* `@stsewd <http://github.com/stsewd>`__: Run coverage on travis (`#4605 <https://github.com/rtfd/readthedocs.org/pull/4605>`__)
+
 Version 2.7.1
 -------------
 

README.rst

@@ -1,7 +1,7 @@
 Welcome to Read the Docs
 ========================
 
-|build-status| |docs|
+|build-status| |docs| |coverage|
 
 Purpose
 -------
@@ -70,6 +70,11 @@ when you push to GitHub.
     :scale: 100%
     :target: https://docs.readthedocs.io/en/latest/?badge=latest
 
+.. |coverage| image:: https://codecov.io/gh/rtfd/readthedocs.org/branch/master/graph/badge.svg
+    :alt: Test coverage
+    :scale: 100%
+    :target: https://codecov.io/gh/rtfd/readthedocs.org
+
 License
 -------
 

contrib/add-project.py

@@ -0,0 +1,168 @@
+"""
+Script to add a Git project as cloned from a Github repository
+to ReadTheDocs. Invoke from the checkout.
+"""
+
+__requires__ = ['requests-toolbelt', 'autocommand', 'keyring']
+
+
+import os
+import getpass
+import re
+import urllib.parse
+import subprocess
+import pathlib
+
+import autocommand
+import keyring
+from requests_toolbelt import sessions
+
+
+rtd = sessions.BaseUrlSession(
+    os.environ.get('RTD_URL', 'https://readthedocs.org/api/v2/'))
+github = sessions.BaseUrlSession('https://api.github.com/')
+github.headers.update(Accept='application/vnd.github.v3+json')
+
+
+class User:
+    """
+    A User (with a password) in RTD.
+
+    Resolves username using ``getpass.getuser()``. Override with
+    RTD_USERNAME environment variable.
+
+    Resolves password using keyring. Install keyring and run
+    ``keyring set https://readthedocs.org/ $USER`` to set the pw.
+    Override with RTD_PASSWORD environment variable.
+    """
+    def __init__(self):
+        self.name = os.environ.get('RTD_USERNAME') or getpass.getuser()
+        system = rtd.create_url('/')
+        self.password = (
+            os.environ.get('RTD_PASSWORD')
+            or keyring.get_password(system, self.name)
+        )
+
+    @property
+    def id(self):
+        resp = rtd.get('../v1/user/', params=dict(username=self.name))
+        resp.raise_for_status()
+        ob, = resp.json()['objects']
+        return ob['id']
+
+    @property
+    def tuple(self):
+        return self.name, self.password
+
+
+class Sluggable(str):
+    """
+    A name for use in RTD with a 'slug' version.
+    """
+    @property
+    def slug(self):
+        return self.replace('.', '')
+
+
+class Repo:
+    """
+    A Git repo
+    """
+
+    def __init__(self, root):
+        self.root = root
+        cmd = ['git', '-C', root, 'remote', 'get-url', 'origin']
+        proc = subprocess.run(
+            cmd, check=True, text=True, stdout=subprocess.PIPE)
+        self.url = proc.stdout.strip()
+
+    @property
+    def name(self):
+        return Sluggable(pathlib.Path(self.url).stem)
+
+
+def create_project(repo):
+    """
+    Create the project from a Repo
+    """
+    user = User()
+    payload = dict(
+        repo=repo.url,
+        slug=repo.name.slug,
+        name=repo.name,
+        users=[user.id],
+    )
+    resp = rtd.post('project/', json=payload, auth=user.tuple)
+    resp.raise_for_status()
+
+
+def configure_github(name, url):
+    """
+    Given a project name and webhook URL, configure the webhook
+    in GitHub.
+
+    Resolves username from ``getpass.getuser()``. Override with
+    ``GITHUB_USERNAME``.
+
+    Resolves access token from keyring for username and system
+    'github.com'. Override with ``GITHUB_TOKEN`` environment
+    variable.
+    """
+    user = os.environ.get('GITHUB_USERNAME') or getpass.getuser()
+    token = (
+        keyring.get_password('github.com', user)
+        or os.environ['GITHUB_TOKEN']
+    )
+    headers = dict(Authorization=f'token {token}')
+    path = f'/repos/{user}/{name}/hooks'
+    params = dict(
+        name='web',
+        config=dict(
+            url=url,
+            content_type='json',
+        ),
+    )
+    github.post(path, json=params, headers=headers)
+
+
+def configure_webhook(name):
+    """
+    Identify the webhook URL for a RTD project named name.
+    """
+    login_path = '/accounts/login/'
+    resp = rtd.get(login_path)
+    token = rtd.cookies.get('csrftoken') or rtd.cookies['csrf']
+    user = User()
+    params = dict(
+        login=user.name,
+        password=user.password,
+        csrfmiddlewaretoken=token,
+        next='/',
+    )
+    headers = dict(Referer=rtd.create_url(login_path))
+    resp = rtd.post(login_path, data=params, headers=headers)
+    token = rtd.cookies.get('csrftoken') or rtd.cookies['csrf']
+    params = dict(
+        integration_type='github_webhook',
+        csrfmiddlewaretoken=token,
+        next='/',
+    )
+    create_path = f'/dashboard/{name.slug}/integrations/create/'
+    headers = dict(
+        Referer=rtd.create_url(create_path),
+    )
+    resp = rtd.post(
+        create_path,
+        data=params,
+        headers=headers,
+    )
+    resp.raise_for_status()
+    ref = re.search(f'<a href="(.*?)">.*?webhook.*?</a>', resp.text).group(1)
+    return urllib.parse.urljoin(resp.url, ref)
+
+
+@autocommand.autocommand(__name__)
+def main(repo: Repo = Repo('.')):
+    create_project(repo)
+    url = configure_webhook(repo.name)
+    configure_github(repo.name, url)

docs/.rstcheck.cfg

@@ -1,4 +1,4 @@
 [rstcheck]
-ignore_directives=automodule,http:get
+ignore_directives=automodule,http:get,tabs,tab
 ignore_roles=djangosetting,setting
 ignore_messages=(Duplicate implicit target name: ".*")|(Hyperlink target ".*" is not referenced)

docs/api/v2.rst

@@ -253,6 +253,7 @@ Build list
     :>json array results: Array of ``Build`` objects.
 
     :query string project__slug: Narrow to builds for a specific ``Project``
+    :query string commit: Narrow to builds for a specific ``commit``
 
 Build detail
 ++++++++++++

docs/builds.rst

@@ -56,6 +56,12 @@ and MkDocs will build the ``.md`` files in that directory.
 As MkDocs doesn't support automatic PDF generation, 
 Read the Docs cannot create a PDF version of your documentation with the *Mkdocs* option.
 
+.. warning::
+
+   We strongly recommend to :ref:`pin the MkDocs version <guides/specifying-dependencies:Specifying Dependencies>`
+   used for your project to build the docs to avoid potential future incompatibilities.
+
+
 Understanding what's going on
 -----------------------------
 

docs/business/analytics.rst renamed to docs/commercial/analytics.rst

@@ -0,0 +1,37 @@
+Custom Domains
+==============
+
+Subdomain support
+-----------------
+
+Once a project is imported under Read the Docs,
+by default it's hosted under a subdomain on one of our domains.
+If you need a custom domain, continue on custom domain setup.
+
+Custom domains
+--------------
+
+Projects can also be hosted under a custom domain.
+If you'd prefer to use your own domain name instead of our default hosting domain,
+you can still host with us.
+
+
+We require two steps from your side:
+
+* Add a CNAME record in your DNS that points to our servers ``<organization-slug>.users.readthedocs.com``
+* Set your project's Privacy Level to *Public* from **Project Admin > Advance Settings**.
+* Add a Domain in the **Project Admin > Domains** page for your project.
+
+.. note:: The domain that should be used is the actual subdomain that you want your docs served on.
+          Generally it will be ``docs.projectname.com``.
+
+
+Custom domain SSL
+-----------------
+
+We do support SSL for CNAMEs on our side.
+Please, `contact our support team`_ to setup it.
+
+.. note:: SSL is required for *private* projects.
+
+.. _contact our support team: mailto:[email protected]

docs/commercial/custom_domains.rst

@@ -1,11 +1,11 @@
-Read the Docs Business Features
--------------------------------
+Read the Docs Commercial Features
+---------------------------------
 
-.. note:: These features are for our new business offering, `readthedocs.com`_.
+.. note:: These features are for our new commercial offering, `readthedocs.com`_.
 
 
 All of the other features outlined in these docs work on both sites.
-Things inside this section are specific to our business offering.
+Things inside this section are specific to our commercial offering.
 
 The largest feature that is different is that documentation on `readthedocs.com`_ is **private**.
 If you have private code that you want documentation for,
@@ -15,6 +15,7 @@ this is our solution.
 
 .. toctree:: 
 
+   custom_domains
    organizations
    sharing
    analytics