Skip to content

Commit ea43371

Browse files
humitoshumitos
authored
Development: use wrangler locally (update NGINX/Dockerfile config) (#10965)
* Development: use `wrangler` locally (update NGINX/Dockerfile config) Related readthedocs/addons#217 * Add CF Worker .js script * Proxy JS files to Addons GitHub's repository * Point common to wrangler branch * Docs: update development install page * Add NGINX cache for addons js files * Add small readme explaining how each NGINX configuration works * update `common/` to point to the latest version
1 parent 607df04 commit ea43371

File tree

7 files changed

+392
-13
lines changed

7 files changed

+392
-13
lines changed

‎common

‎dockerfiles/Dockerfile.wrangler

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
FROM node:18.15
2+
RUN npm install -g [email protected]

‎dockerfiles/force-readthedocs-addons.js

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
/*
2+
3+
Script to inject the new Addons implementation on pages served by El Proxito.
4+
5+
This script is ran on a Cloudflare Worker and modifies the HTML with two different purposes:
6+
7+
1. remove the old implementation of our flyout (``readthedocs-doc-embed.js`` and others)
8+
2. inject the new addons implementation (``readthedocs-addons.js``) script
9+
10+
Currently, we are doing 1) only when users opt-in into the new beta addons.
11+
In the future, when our addons become stable, we will always remove the old implementation,
12+
making all the projects to use the addons by default.
13+
14+
*/
15+
16+
// add "readthedocs-addons.js" inside the "<head>"
17+
const addonsJs =
18+
'<script async type="text/javascript" src="/_/static/javascript/readthedocs-addons.js"></script>';
19+
20+
// selectors we want to remove
21+
// https://developers.cloudflare.com/workers/runtime-apis/html-rewriter/#selectors
22+
const analyticsJs =
23+
'script[src="/_/static/javascript/readthedocs-analytics.js"]';
24+
const docEmbedCss = 'link[href="/_/static/css/readthedocs-doc-embed.css"]';
25+
const docEmbedJs =
26+
'script[src="/_/static/javascript/readthedocs-doc-embed.js"]';
27+
const analyticsJsAssets =
28+
'script[src="https://assets.readthedocs.org/static/javascript/readthedocs-analytics.js"]';
29+
const docEmbedCssAssets =
30+
'link[href="https://assets.readthedocs.org/static/css/readthedocs-doc-embed.css"]';
31+
const docEmbedJsAssets =
32+
'script[src="https://assets.readthedocs.org/static/javascript/readthedocs-doc-embed.js"]';
33+
const docEmbedJsAssetsCore =
34+
'script[src="https://assets.readthedocs.org/static/core/js/readthedocs-doc-embed.js"]';
35+
const badgeOnlyCssAssets =
36+
'link[href="https://assets.readthedocs.org/static/css/badge_only.css"]';
37+
const badgeOnlyCssAssetsProxied = 'link[href="/_/static/css/badge_only.css"]';
38+
const readthedocsExternalVersionWarning = "[role=main] > div:first-child > div:first-child.admonition.warning";
39+
const readthedocsFlyout = "div.rst-versions";
40+
41+
// "readthedocsDataParse" is the "<script>" that calls:
42+
//
43+
// READTHEDOCS_DATA = JSON.parse(document.getElementById('READTHEDOCS_DATA').innerHTML);
44+
//
45+
const readthedocsDataParse = "script[id=READTHEDOCS_DATA]:first-of-type";
46+
const readthedocsData = "script[id=READTHEDOCS_DATA]";
47+
48+
// do this on a fetch
49+
addEventListener("fetch", (event) => {
50+
const request = event.request;
51+
event.respondWith(handleRequest(request));
52+
});
53+
54+
async function handleRequest(request) {
55+
// perform the original request
56+
let originalResponse = await fetch(request);
57+
58+
// get the content type of the response to manipulate the content only if it's HTML
59+
const contentType = originalResponse.headers.get("content-type") || "";
60+
const injectHostingIntegrations =
61+
originalResponse.headers.get("x-rtd-hosting-integrations") || "false";
62+
const forceAddons =
63+
originalResponse.headers.get("x-rtd-force-addons") || "false";
64+
65+
// Log some debugging data
66+
console.log(`ContentType: ${contentType}`);
67+
console.log(`X-RTD-Force-Addons: ${forceAddons}`);
68+
console.log(`X-RTD-Hosting-Integrations: ${injectHostingIntegrations}`);
69+
70+
// get project/version slug from headers inject by El Proxito
71+
const projectSlug = originalResponse.headers.get("x-rtd-project") || "";
72+
const versionSlug = originalResponse.headers.get("x-rtd-version") || "";
73+
74+
// check to decide whether or not inject the new beta addons:
75+
//
76+
// - content type has to be "text/html"
77+
// when all these conditions are met, we remove all the old JS/CSS files and inject the new beta flyout JS
78+
79+
// check if the Content-Type is HTML, otherwise do nothing
80+
if (contentType.includes("text/html")) {
81+
// Remove old implementation of our flyout and inject the new addons if the following conditions are met:
82+
//
83+
// - header `X-RTD-Force-Addons` is present (user opted-in into new beta addons)
84+
// - header `X-RTD-Hosting-Integrations` is not present (added automatically when using `build.commands`)
85+
//
86+
if (forceAddons === "true" && injectHostingIntegrations === "false") {
87+
return (
88+
new HTMLRewriter()
89+
.on(analyticsJs, new removeElement())
90+
.on(docEmbedCss, new removeElement())
91+
.on(docEmbedJs, new removeElement())
92+
.on(analyticsJsAssets, new removeElement())
93+
.on(docEmbedCssAssets, new removeElement())
94+
.on(docEmbedJsAssets, new removeElement())
95+
.on(docEmbedJsAssetsCore, new removeElement())
96+
.on(badgeOnlyCssAssets, new removeElement())
97+
.on(badgeOnlyCssAssetsProxied, new removeElement())
98+
.on(readthedocsExternalVersionWarning, new removeElement())
99+
.on(readthedocsFlyout, new removeElement())
100+
// NOTE: I wasn't able to reliably remove the "<script>" that parses
101+
// the "READTHEDOCS_DATA" defined previously, so we are keeping it for now.
102+
//
103+
// .on(readthedocsDataParse, new removeElement())
104+
// .on(readthedocsData, new removeElement())
105+
.on("head", new addPreloads())
106+
.on("head", new addProjectVersionSlug(projectSlug, versionSlug))
107+
.transform(originalResponse)
108+
);
109+
}
110+
111+
// Inject the new addons if the following conditions are met:
112+
//
113+
// - header `X-RTD-Hosting-Integrations` is present (added automatically when using `build.commands`)
114+
// - header `X-RTD-Force-Addons` is not present (user opted-in into new beta addons)
115+
//
116+
if (forceAddons === "false" && injectHostingIntegrations === "true") {
117+
return new HTMLRewriter()
118+
.on("head", new addPreloads())
119+
.on("head", new addProjectVersionSlug(projectSlug, versionSlug))
120+
.transform(originalResponse);
121+
}
122+
}
123+
124+
// Modify `_static/searchtools.js` to re-enable Sphinx's default search
125+
if (
126+
(contentType.includes("text/javascript") ||
127+
contentType.includes("application/javascript")) &&
128+
(injectHostingIntegrations === "true" || forceAddons === "true") &&
129+
originalResponse.url.endsWith("_static/searchtools.js")
130+
) {
131+
console.log("Modifying _static/searchtools.js");
132+
return handleSearchToolsJSRequest(originalResponse);
133+
}
134+
135+
// if none of the previous conditions are met,
136+
// we return the response without modifying it
137+
return originalResponse;
138+
}
139+
140+
class removeElement {
141+
element(element) {
142+
console.log("Removing: " + element.tagName);
143+
console.log("Attribute href: " + element.getAttribute("href"));
144+
console.log("Attribute src: " + element.getAttribute("src"));
145+
console.log("Attribute id: " + element.getAttribute("id"));
146+
console.log("Attribute class: " + element.getAttribute("class"));
147+
element.remove();
148+
}
149+
}
150+
151+
class addPreloads {
152+
element(element) {
153+
console.log("addPreloads");
154+
element.append(addonsJs, { html: true });
155+
}
156+
}
157+
158+
class addProjectVersionSlug {
159+
constructor(projectSlug, versionSlug) {
160+
this.projectSlug = projectSlug;
161+
this.versionSlug = versionSlug;
162+
}
163+
164+
element(element) {
165+
console.log(
166+
`addProjectVersionSlug. projectSlug=${this.projectSlug} versionSlug=${this.versionSlug}`,
167+
);
168+
if (this.projectSlug && this.versionSlug) {
169+
const metaProject = `<meta name="readthedocs-project-slug" content="${this.projectSlug}" />`;
170+
const metaVersion = `<meta name="readthedocs-version-slug" content="${this.versionSlug}" />`;
171+
172+
element.append(metaProject, { html: true });
173+
element.append(metaVersion, { html: true });
174+
}
175+
}
176+
}
177+
178+
/*
179+
180+
Script to fix the old removal of the Sphinx search init.
181+
182+
Enabling addons breaks the default Sphinx search in old versions that are not possible to rebuilt.
183+
This is because we solved the problem in the `readthedocs-sphinx-ext` extension,
184+
but since those versions can't be rebuilt, the fix does not apply there.
185+
186+
To solve the problem in these old versions, we are using a CF worker to apply that fix on-the-fly
187+
at serving time on those old versions.
188+
189+
The fix basically replaces a Read the Docs comment in file `_static/searchtools.js`,
190+
introduced by `readthedocs-sphinx-ext` to _disable the initialization of Sphinx search_,
191+
with the real JavaScript to initialize the search, as Sphinx does by default.
192+
(in other words, it _reverts_ the manipulation done by `readthedocs-sphinx-ext`)
193+
194+
*/
195+
196+
const textToReplace = `/* Search initialization removed for Read the Docs */`;
197+
const textReplacement = `
198+
/* Search initialization manipulated by Read the Docs using Cloudflare Workers */
199+
/* See https://github.com/readthedocs/addons/issues/219 for more information */
200+
201+
function initializeSearch() {
202+
Search.init();
203+
}
204+
205+
if (document.readyState !== "loading") {
206+
initializeSearch();
207+
}
208+
else {
209+
document.addEventListener("DOMContentLoaded", initializeSearch);
210+
}
211+
`;
212+
213+
async function handleSearchToolsJSRequest(originalResponse) {
214+
const content = await originalResponse.text();
215+
const modifiedResponse = new Response(
216+
content.replace(textToReplace, textReplacement),
217+
);
218+
return modifiedResponse;
219+
}

‎dockerfiles/nginx/README.rst

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
How NGINX proxy works
2+
=====================
3+
4+
Read the Docs uses 3 different NGINX configurations;
5+
6+
web
7+
This configuration is in charge of serving the dashboard application
8+
on ``$NGINX_WEB_SERVER_NAME`` domain.
9+
It listens at port 80 and proxies it to ``web`` container on port ``8000``,
10+
where is the Django application running.
11+
12+
It also proxies assets files under ``/static/`` to the ``storage`` container
13+
on port ``9000`` which is running MinIO (S3 emulator).
14+
15+
proxito
16+
Its main goal is to serve documentation pages and handle 404s.
17+
It proxies all the requests to ``proxito`` container on port ``8000``,
18+
where the "El Proxito" Django application is running.
19+
This application returns a small response with ``X-Accel-Redirect`` special HTTP header
20+
pointing to a MinIO (S3 emulator) path which is used by NGINX to proxy to it.
21+
22+
Besides, the response from El Proxito contains a bunch of HTTP headers
23+
that are added by NGINX to the MinIO response to end up in the resulting
24+
response arriving to the user.
25+
26+
It also configures a 404 fallback that hits an internal URL on the
27+
Django application to handle them correctly
28+
(redirects and custom 404 pages, among others)
29+
30+
Finally, there are two special URLs configured to proxy the JavaScript files
31+
required for Read the Docs Addons and serve them directly from a GitHub tag.
32+
33+
Note server is not exposed _outside_ the Docker internal's network,
34+
and is accessed only via Wrangler. Keep reading to understand how it's connected.
35+
36+
wrangler
37+
Node.js implementation of Cloudflare Worker that's in front of "El Proxito".
38+
It's listening on ``$NGINX_PROXITO_SERVER_NAME`` domain and executes the worker
39+
``force-readthedocs-addons.js``.
40+
41+
This worker hits ``proxito`` NGINX server listening at ``nginx`` container
42+
on port ``8080`` to fetch the "original response" and manipulates it to
43+
inject extra HTTP tags required for Read the Docs Addons (``meta`` and ``script``).
44+
45+
46+
47+
ASCII-art explanation
48+
---------------------
49+
50+
.. I used: https://asciiflow.com/
51+
52+
53+
Documentation page on ``$NGINX_PROXITO_SERVER_NAME``
54+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
55+
56+
.. text::
57+
58+
┌──────────────── User
59+
│
60+
│ ▲
61+
(documentation pages) │
62+
│ │
63+
│ │
64+
▼ 80 │
65+
┌────────────────┐ │
66+
│ │ │
67+
│ │ │
68+
│ │ │
69+
│ wrangler │ ──────────┘
70+
│ │
71+
│ │
72+
│ │
73+
└──────┬─────────┘ ┌──────────────┐ ┌────────────────┐
74+
│ ▲ │ │ │ │
75+
│ │ │ │ 9000│ │
76+
│ └──────────────────── │ ├───────►│ │
77+
│ │ NGINX │ │ MinIO (S3) │
78+
└───────────────────────► │ │◄───────┤ │
79+
8080 │ │ │ │
80+
│ │ │ │
81+
└─────┬────────┘ └────────────────┘
82+
│ ▲
83+
│ │
84+
│ │
85+
│ │
86+
8000 ▼ │
87+
┌────────┴─────┐
88+
│ │
89+
│ │
90+
│ El Proxito │
91+
│ │
92+
│ │
93+
└──────────────┘
94+
95+
96+
Documentation page on ``$NGINX_WEB_SERVER_NAME``
97+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
98+
99+
100+
.. text::
101+
102+
User
103+
104+
│
105+
│
106+
(dashboard)
107+
│
108+
│
109+
│
110+
▼ 80
111+
┌──────────────┐ ┌────────────────┐
112+
│ │ │ │
113+
│ │ 9000│ │
114+
│ ├───────►│ │
115+
│ NGINX │ │ MinIO (S3) │
116+
│ │◄───────┤ │
117+
│ │ │ │
118+
│ │ │ │
119+
└─────┬────────┘ └────────────────┘
120+
│ ▲
121+
│ │
122+
│ │
123+
│ │
124+
8000 ▼ │
125+
┌────────┴─────┐
126+
│ │
127+
│ │
128+
│ web │
129+
│ │
130+
│ │
131+
└──────────────┘

0 commit comments

Comments
 (0)