Skip to content

Commit dd12195

Browse files
ericholscherericholscher
committed
Merge remote-tracking branch 'origin/master' into improve-search-admin
2 parents d7a023b + 0496f9a commit dd12195

File tree

9 files changed

+374
-26
lines changed

9 files changed

+374
-26
lines changed

‎.gitattributes

+1
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,4 @@
1313
*.whl binary
1414
*.woff binary
1515
*.woff2 binary
16+
*.gif binary

‎CHANGELOG.rst

+6
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,12 @@ This is a quick hotfix to the previous version.
1111
Version 3.5.1
1212
-------------
1313

14+
This version contained a `security fix <https://github.com/rtfd/readthedocs.org/security/advisories/GHSA-2mw9-4c46-qrcv>`_
15+
for an open redirect issue.
16+
The problem has been fixed and deployed on readthedocs.org.
17+
For users who depend on the Read the Docs code line for a private instance of Read the Docs,
18+
you are encouraged to update to 3.5.1 as soon as possible.
19+
1420
:Date: June 11, 2019
1521

1622
* `@stsewd <http://github.com/stsewd>`__: Update build images in docs (`#5782 <https://github.com/rtfd/readthedocs.org/pull/5782>`__)

‎docs/_static/images/design-docs/in-doc-search-ui/in-doc-search-ui-demo.gif

1.82 MB
Loading

‎docs/design/in-doc-search-ui.rst

+183
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
In Doc Search UI
2+
================
3+
4+
Giving readers the ability to easily search the information
5+
that they are looking for is important for us.
6+
We have already upgraded to the latest version of `Elasticsearch`_ and
7+
we plan to implement `search as you type` feature for all the documentations hosted by us.
8+
It will be designed to provide instant results as soon as the user starts
9+
typing in the search bar with a clean and minimal frontend.
10+
This design document aims to provides the details of it.
11+
This is a GSoC'19 project.
12+
13+
.. warning::
14+
15+
This design document details future features that are **not yet implemented**.
16+
To discuss this document, please get in touch in the `issue tracker`_.
17+
18+
19+
The final result may look something like this:
20+
21+
.. figure:: ../_static/images/design-docs/in-doc-search-ui/in-doc-search-ui-demo.gif
22+
:align: center
23+
:target: ../_static/images/design-docs/in-doc-search-ui/in-doc-search-ui-demo.gif
24+
25+
Short demo
26+
27+
28+
Goals And Non-Goals
29+
-------------------
30+
31+
Project Goals
32+
++++++++++++++
33+
34+
* Support a search-as-you-type/autocomplete interface.
35+
* Support across all (or virtually all) Sphinx themes.
36+
* Support for the JavaScript user experience down to IE11 or graceful degradation where we can't support it.
37+
* Project maintainers should have a way to opt-in/opt-out of this feature.
38+
* (Optional) Project maintainers should have the flexibility to change some of the styles using custom CSS and JS files.
39+
40+
Non-Goals
41+
++++++++++
42+
43+
* For the initial release, we are targeting only Sphinx documentations
44+
as we don't index MkDocs documentations to our Elasticsearch index.
45+
46+
47+
Existing Search Implementation
48+
------------------------------
49+
50+
We have a detailed documentation explaing the underlying architecture of our search backend
51+
and how we index documents to our Elasticsearch index.
52+
You can read about it :doc:`here <../development/search>`.
53+
54+
55+
Proposed Architecture for In-Doc Search UI
56+
------------------------------------------
57+
58+
Frontend
59+
++++++++
60+
61+
Technologies
62+
~~~~~~~~~~~~
63+
64+
Frontend is to designed in a theme agnostics way. For that,
65+
we explored various libraries which may be of use but none of them fits our needs.
66+
So, we might be using vanilla JavaScript for this purpose.
67+
This will provide us some advantages over using any third party library:
68+
69+
* Better control over the DOM.
70+
* Performance benefits.
71+
72+
73+
Proposed Architecture
74+
~~~~~~~~~~~~~~~~~~~~~
75+
76+
We plan to select the search bar, which is present in every theme,
77+
and use the `querySelector()`_ method of JavaScript.
78+
Then add an event listener to it to listen for the changes and
79+
fire a search query to our backend as soon as there is any change.
80+
Our backend will then return the suggestions,
81+
which will be shown to the user in a clean and minimal UI.
82+
We will be using `document.createElement()`_ and `node.removeChild()`_ method
83+
provided by JavaScript as we don't want empty `<div>` hanging out in the DOM.
84+
85+
We have a few ways to include the required JavaScript and CSS files in all the projects:
86+
87+
* Add CSS into `readthedocs-doc-embed.css` and JS into `readthedocs-doc-embed.js`
88+
and it will get included.
89+
* Package the in-doc search into it's own self-contained CSS and JS files
90+
and include them in a similar manner to `readthedocs-doc-embed.*`.
91+
* It might be possible to package up the in-doc CSS/JS as a sphinx extension.
92+
This might be nice because then it's easy to enable it on a per-project basis.
93+
When we are ready to roll it out to a wider audience,
94+
we can make a decision to just turn it on for everybody (put it in `here`_)
95+
or we could enable it as an opt-in feature like the `404 extension`_.
96+
97+
98+
UI/UX
99+
~~~~~
100+
101+
We have two ways which can be used to show suggestions to the user.
102+
103+
* Show suggestions below the search bar.
104+
* Open a full page search interface when the user click on search field.
105+
106+
107+
Backend
108+
+++++++
109+
110+
We have a few options to support `search as you type` feature,
111+
but we need to decide that which option would be best for our use-case.
112+
113+
Edge NGram Tokenizer
114+
~~~~~~~~~~~~~~~~~~~~
115+
116+
* Pros
117+
118+
* More effective than Completion Suggester when it comes to autocompleting
119+
words that can appear in any order.
120+
* It is considerable fast because most of the work is being done at index time,
121+
hence the time taken for autocompletion is reduced.
122+
* Supports highlighting of the matching terms.
123+
124+
* Cons
125+
126+
* Requires greater disk space.
127+
128+
129+
Completion Suggester
130+
~~~~~~~~~~~~~~~~~~~~
131+
132+
* Pros
133+
134+
* Really fast as it is optimized for speed.
135+
* Does not require large disk space.
136+
137+
* Cons
138+
139+
* Matching always starts at the beginning of the text. So, for example,
140+
"Hel" will match "Hello, World" but not "World Hello".
141+
* Highlighting of the matching words is not supported.
142+
* According to the official docs for Completion Suggester,
143+
fast lookups are costly to build and are stored in-memory.
144+
145+
146+
Milestones
147+
----------
148+
149+
+-----------------------------------------------------------------------------------+------------------+
150+
| Milestone | Due Date |
151+
+===================================================================================+==================+
152+
| A local implementation of the project. | 12th June, 2019 |
153+
+-----------------------------------------------------------------------------------+------------------+
154+
| In-doc search on a test project hosted on Read the Docs using the RTD Search API. | 20th June, 2019 |
155+
+-----------------------------------------------------------------------------------+------------------+
156+
| In-doc search on docs.readthedocs.io. | 20th June, 2019 |
157+
+-----------------------------------------------------------------------------------+------------------+
158+
| Friendly user trial where users can add this on their own docs. | 5th July, 2019 |
159+
+-----------------------------------------------------------------------------------+------------------+
160+
| Additional UX testing on the top-10 Sphinx themes. | 15th July, 2019 |
161+
+-----------------------------------------------------------------------------------+------------------+
162+
| Finalize the UI. | 25th July, 2019 |
163+
+-----------------------------------------------------------------------------------+------------------+
164+
| Improve the search backend for efficient and fast search results. | 10th August, 2019|
165+
+-----------------------------------------------------------------------------------+------------------+
166+
167+
168+
Open Questions
169+
++++++++++++++
170+
171+
* Should we rely on jQuery, any third party library or pure vanilla JavaScript?
172+
* Are the subprojects to be searched?
173+
* Is our existing Search API is sufficient?
174+
* Should we go for edge ngrams or completion suggester?
175+
176+
177+
.. _issue tracker: https://github.com/rtfd/readthedocs.org/issues
178+
.. _Elasticsearch: https://www.elastic.co/products/elasticsearch
179+
.. _querySelector(): https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector
180+
.. _document.createElement(): https://developer.mozilla.org/en-US/docs/Web/API/Document/createElement
181+
.. _node.removeChild(): https://developer.mozilla.org/en-US/docs/Web/API/Node/removeChild
182+
.. _here: https://github.com/rtfd/readthedocs.org/blob/9ca5858e859dea0759d913e8db70a623d62d6a16/readthedocs/doc_builder/templates/doc_builder/conf.py.tmpl#L135-L142
183+
.. _404 extension : https://github.com/rtfd/sphinx-notfound-page

‎docs/design/pr-builder.rst

+153
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
Design of Pull Request Builder
2+
==============================
3+
4+
Background
5+
----------
6+
7+
This will focus on automatically building documentation for Pull Requests on Read the Docs projects.
8+
This is one of the most requested feature of Read the Docs.
9+
This document will serve as a design document for discussing how to implement this features.
10+
11+
Scope
12+
-----
13+
14+
- Making Pull Requests work like temporary ``Version``
15+
- Excluding PR Versions from Elasticsearch Indexing
16+
- Adding a ``PR Builds`` Tab in the Project Dashboard
17+
- Updating the Footer API
18+
- Adding Warning Banner to Docs
19+
- Serving PR Docs
20+
- Excluding PR Versions from Search Engines
21+
- Receiving ``pull_request`` webhook event from Github
22+
- Fetching data from pull requests
23+
- Storing PR Version build Data
24+
- Creating PR Versions when a pull request is opened and Triggering a build
25+
- Triggering Builds on new commits on a PR
26+
- Status reporting to Github
27+
28+
Fetching Data from Pull Requests
29+
--------------------------------
30+
31+
We already get Pull request events from Github webhooks.
32+
We can utilize that to fetch data from pull requests.
33+
when a ``pull_request`` event is triggered we can fetch the data of that pull request.
34+
We can fetch the pull request by doing something similar to travis-ci.
35+
ie: ``git fetch origin +refs/pull/<pr_number>/merge:``
36+
37+
Modeling Pull Requests as a Type of Version
38+
-------------------------------------------
39+
40+
Pull requests can be Treated as a Type of Temporary ``Version``.
41+
We might consider adding a ``VERSION_TYPES`` to the ``Version`` model.
42+
43+
- If we go with ``VERSION_TYPES`` we can add something like ``pull_request`` alongside Tag and Branch.
44+
45+
We should add ``Version`` and ``Build`` Model Managers for PR and Regular Versions and Builds.
46+
The proposed names for PR and Regular Version and Build Mangers are ``external`` and ``internal``.
47+
48+
We can then use ``Version.internal.all()`` to get all regular versions,
49+
``Version.external.all()`` to get all PR versions.
50+
51+
We can then use ``Build.internal.all()`` to get all regular version builds,
52+
``Build.external.all()`` to get all PR version builds.
53+
54+
55+
Excluding PR Versions from Elasticsearch Indexing
56+
-------------------------------------------------
57+
58+
We should exclude to PR Versions from being Indexed to Elasticsearch.
59+
We need to update the queryset to exclude PR Versions.
60+
61+
Adding a PR Builds Tab in the Project Dashboard
62+
-----------------------------------------------
63+
64+
We can add a Tab in the project dashboard that will listout the PR Builds of that project.
65+
We can name it ``PR Builds``.
66+
67+
Creating Versions for Pull Requests
68+
-----------------------------------
69+
70+
If the Github webhook event is ``pull_request`` and action is ``opened``,
71+
this means a pull request was opened in the projects repository.
72+
We can create a ``Version`` from the Payload data and trigger a initial build for the version.
73+
A version will be created whenever RTD receives an event like this.
74+
75+
Triggering Build for New Commits in a Pull Request
76+
--------------------------------------------------
77+
78+
We might want to trigger a new build for the PR version if there is a new commit on the PR.
79+
If the Github webhook event is ``pull_request`` and action is ``synchronize``,
80+
this means a new commit was added to the pull request.
81+
82+
Status Reporting to Github
83+
--------------------------
84+
85+
We could send build status reports to Github. We could send if the build was Successful or Failed.
86+
We can also send the build URL. By this we could show if the build passed or failed on Github something like travis-ci does.
87+
88+
As we already have the ``repo:status`` scope on our OAuth App,
89+
we can send the status report to Github using the Github Status API.
90+
91+
Sending the status report would be something like this:
92+
93+
.. http:post:: /repos/:owner/:repo/statuses/:sha
94+
95+
.. code:: json
96+
97+
{
98+
"state": "success",
99+
"target_url": "<pr_build_url>",
100+
"description": "The build succeeded!",
101+
"context": "continuous-documentation/read-the-docs"
102+
}
103+
104+
Storing Pull Request Docs
105+
-------------------------
106+
107+
We need to think about how and where to store data after a PR Version build is finished.
108+
We can store the data in a blob storage.
109+
110+
Excluding PR Versions from Search Engines
111+
-----------------------------------------
112+
113+
We should Exclude the PR Versions from Search Engines,
114+
because it might cause problems for RTD users.
115+
As users might land to a pull request doc but not the original Project Docs.
116+
This will cause confusion for the users.
117+
118+
Serving PR Docs
119+
---------------
120+
121+
We need to think about how we want to serve the PR Docs.
122+
123+
- We could serve the PR Docs from another Domain.
124+
- We could serve the PR Docs using ``<pr_number>`` namespace on the same Domain.
125+
126+
- Using ``pr-<pr_number>`` as the version slug ``https://<project_slug>.readthedocs.io/<language_code>/pr-<pr_number>/``
127+
- Using ``pr`` subdomain ``https://pr.<project_slug>.readthedocs.io/<pr_number>/``
128+
129+
130+
Updating the Footer API
131+
-----------------------
132+
133+
We need to update the Footer API to reflect the changes.
134+
We might want to have a way to show that if this is a PR Build on the Footer.
135+
136+
- For regular project docs we should remove the PR Versions from the version list of the Footer.
137+
- We might want to send ``is_pr`` data with the Footer API response.
138+
139+
Adding Warning Banner to Docs
140+
-----------------------------
141+
142+
We need to add a warning banner to the PR Version Docs to let the users know that this is a Draft/PR version.
143+
We can use a sphinx extension that we will force to install on the PR Versions to add the warning banner.
144+
145+
Related Issues
146+
--------------
147+
148+
- `Autobuild Docs for Pull Requests`_
149+
- `Add travis-ci style pull request builder`_
150+
151+
152+
.. _Autobuild Docs for Pull Requests: https://github.com/rtfd/readthedocs.org/issues/5684
153+
.. _Add travis-ci style pull request builder: https://github.com/rtfd/readthedocs.org/issues/1340

‎docs/security.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ Security issue archive
8787
Version 3.5.1
8888
~~~~~~~~~~~~~
8989

90-
Version 3.5.1 fixed an issue where that affected projects with "prefix" or "sphinx" user-defined redirects.
90+
:ref:`changelog:Version 3.5.1` fixed an issue that affected projects with "prefix" or "sphinx" user-defined redirects.
9191
The issue allowed the creation of hyperlinks that looked like they would go to a documentation domain
9292
on Read the Docs (either ``*.readthedocs.io`` or a custom docs domain) but instead went to a different domain.
9393

‎readthedocs/doc_builder/backends/mkdocs.py

+5
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,11 @@ def load_yaml_config(self):
9797
return config
9898

9999
except IOError:
100+
log.info(
101+
'Creating default MkDocs config file for project: %s:%s',
102+
self.project.slug,
103+
self.version.slug,
104+
)
100105
return {
101106
'site_name': self.version.project.name,
102107
}

‎readthedocs/doc_builder/backends/sphinx.py

+5
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,11 @@ def __init__(self, *args, **kwargs):
6060

6161
def _write_config(self, master_doc='index'):
6262
"""Create ``conf.py`` if it doesn't exist."""
63+
log.info(
64+
'Creating default Sphinx config file for project: %s:%s',
65+
self.project.slug,
66+
self.version.slug,
67+
)
6368
docs_dir = self.docs_dir()
6469
conf_template = render_to_string(
6570
'sphinx/conf.py.conf',

0 commit comments

Comments
 (0)