Skip to content

Commit e4ecaa9

Browse files
simonwsimonw
authored
Updating stable docs in ReadTheDocs without pushing a release
Refs: - simonw/simonw-readthedocs-experiments#1 - simonw/simonw-readthedocs-experiments#2
1 parent 11b7ebb commit e4ecaa9

File tree

1 file changed

+266
-0
lines changed

1 file changed

+266
-0
lines changed

‎readthedocs/stable-docs.md

+266
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# Updating stable docs in ReadTheDocs without pushing a release
2+
3+
I use [ReadTheDocs](https://readthedocs.org/) for several of my projects. It's fantastic: among other things, it makes it easy to publish the documentation for my latest `main` branch at `/latest/` and the documentation for my latest release at `/stable/` (as well as maintain archived tag URLs for every prior release).
4+
5+
I can then configure the main page of my project's documentation to redirect to `/stable/` by default.
6+
7+
I'm using it for the following documentation sites:
8+
9+
- https://docs.datasette.io/
10+
- https://sqlite-utils.datasette.io/
11+
- https://llm.datasette.io/
12+
- https://shot-scraper.datasette.io/
13+
14+
And quite a few more.
15+
16+
## The problem: typo fixes, project news and plugins
17+
18+
There's a catch: by default, the only way to update the `/stable/` documentation is to ship a new release.
19+
20+
In the past, I've shipped `x.x.1` version bumps just to get new documentation published to ReadTheDocs!
21+
22+
This isn't great though, especially now I have some of my packages in Homebrew. Shipping a release of Datasette or `sqlite-utils` or LLM means the Homebrew formula has to be updated by someone too, which feels like a waste of time and effort if the only change was to the documentation.
23+
24+
Another challenge is that there are things I want to include in my documentation that aren't actually coupled to releases. Project news is one example, but a better one is plugin listings: when a new plugin is released I'd like to include it in the official documentation, but it's not a code change that justifies pushing a new release.
25+
26+
## The shape of the solution
27+
28+
What I really want is a way to trigger and publish a new build of the `/stable/` documentation on ReadTheDocs without having to ship a new release.
29+
30+
After some [extensive experimentation](https://github.com/simonw/simonw-readthedocs-experiments/issues/1) (that's an issue thread with 43 comments, all by me) I've found a solution.
31+
32+
The basic shape is this: rather than having ReadTheDocs serve `/stable/` from the latest tagged release of my project, I instead maintain a `stable` branch in the GitHub repository. It's this branch that becomes the default documentation on my documentation sites.
33+
34+
Then I use GitHub Actions to automate the process of updating that branch. In particular:
35+
36+
- Any time I tag and push a new release, the `stable` branch is entirely updated to reflect the content of that release. Any changes in that branch are discarded.
37+
- But... I can make edits to that branch myself in between releases. Those edits will be wiped out at the next release, so I need to be sure to apply them to the `main` branch as well.
38+
- I also have a shortcut: any time I commit to `main` I can include the text `!stable-docs` in my commit message. If I do that, GitHub Actions will copy the exact content of any files in the `docs/` directory and use them to update the `stable` branch, then publish that branch with those new changes.
39+
40+
For general usage I only have to do two things: continue to ship releases, and occasionally include `!stable-docs` in a commit that updates a document which I'd like to be reflected instantly on the documentation site (a typo fix, project news or new plugin for example).
41+
42+
## The GitHub Actions workflow
43+
44+
Here's the full `.github/workflows/stable-docs.yml` workflow file I built to implement this process:
45+
46+
```yaml
47+
name: Update Stable Docs
48+
49+
on:
50+
release:
51+
types: [published]
52+
push:
53+
branches:
54+
- main
55+
56+
permissions:
57+
contents: write
58+
59+
jobs:
60+
update_stable_docs:
61+
runs-on: ubuntu-latest
62+
steps:
63+
- name: Checkout repository
64+
uses: actions/checkout@v3
65+
with:
66+
fetch-depth: 0 # We need all commits to find docs/ changes
67+
- name: Set up Git user
68+
run: |
69+
git config user.name "Automated"
70+
git config user.email "[email protected]"
71+
- name: Create stable branch if it does not yet exist
72+
run: |
73+
if ! git ls-remote --heads origin stable | grep stable; then
74+
git checkout -b stable
75+
# If there are any releases, copy docs/ in from most recent
76+
LATEST_RELEASE=$(git tag | sort -Vr | head -n1)
77+
if [ -n "$LATEST_RELEASE" ]; then
78+
rm -rf docs/
79+
git checkout $LATEST_RELEASE -- docs/
80+
fi
81+
git commit -m "Populate docs/ from $LATEST_RELEASE" || echo "No changes"
82+
git push -u origin stable
83+
fi
84+
- name: Handle Release
85+
if: github.event_name == 'release'
86+
run: |
87+
git fetch --all
88+
git checkout stable
89+
git reset --hard ${GITHUB_REF#refs/tags/}
90+
git push origin stable --force
91+
- name: Handle Commit to Main
92+
if: contains(github.event.head_commit.message, '!stable-docs')
93+
run: |
94+
git fetch origin
95+
git checkout -b stable origin/stable
96+
# Get the list of modified files in docs/ from the current commit
97+
FILES=$(git diff-tree --no-commit-id --name-only -r ${{ github.sha }} -- docs/)
98+
# Check if the list of files is non-empty
99+
if [[ -n "$FILES" ]]; then
100+
# Checkout those files to the stable branch to over-write with their contents
101+
for FILE in $FILES; do
102+
git checkout ${{ github.sha }} -- $FILE
103+
done
104+
git add docs/
105+
git commit -m "Doc changes from ${{ github.sha }}"
106+
git push origin stable
107+
else
108+
echo "No changes to docs/ in this commit."
109+
exit 0
110+
fi
111+
```
112+
There are three interesting step blocks here.
113+
114+
### Creating the stable branch
115+
116+
The first block is there purely to create that `stable` branch if it does not exist yet. This means I can drop the above workflow into a new project without having to do any additional setup against the repo:
117+
118+
```yaml
119+
- name: Create stable branch if it does not yet exist
120+
run: |
121+
if ! git ls-remote --heads origin stable | grep stable; then
122+
git checkout -b stable
123+
# If there are any releases, copy docs/ in from most recent
124+
LATEST_RELEASE=$(git tag | sort -Vr | head -n1)
125+
if [ -n "$LATEST_RELEASE" ]; then
126+
rm -rf docs/
127+
git checkout $LATEST_RELEASE -- docs/
128+
fi
129+
git commit -m "Populate docs/ from $LATEST_RELEASE" || echo "No changes"
130+
git push -u origin stable
131+
fi
132+
```
133+
There's another trick in there though. When I add this workflow to a new repository I'm fine for that `stable` branch to start off directly reflecting `main`.
134+
135+
But... if I add it to a repository that already has releases, I need the `stable` branch to start out reflecting the documentation in that most recent release.
136+
137+
That's what the `LATEST_RELEASE` variable is for. `git tag` outputs the list of tags as a newline-separated list. Piping them through `sort` and `head -n1` gives the highest release tag:
138+
139+
```bash
140+
git tag | sort -Vr | head -n1
141+
```
142+
`sort -Vr` causes `sort` to sort the tags using [version sort](https://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html#sort-invocation) in reverse. Here's how that `-V` option is described:
143+
144+
> It behaves like a standard sort, except that each sequence of decimal digits is treated numerically as an index/version number.
145+
146+
Getting the commit that creates the new branch to work was [surprisingly tricky](https://github.com/simonw/simonw-readthedocs-experiments/issues/2)!
147+
148+
The problem is that GitHub Actions has a rule that a workflow is not allowed to modify its own YAML configuration and then push those changes back up to GitHub.
149+
150+
My first version of this worked by creating the new `stable` branch from the most recent tagged release. But... this carries the risk of that tagged version including a change to the workflow YAML, which results in an error.
151+
152+
Eventually I realized that I only care about the contents of that `docs/` directory, so instead of creating the branch from the release `tag` I could instead create the branch from `main` and then copy in the `docs/` directory from that tag.
153+
154+
This avoids any chance of a file in `.github/workflows` being updated in a way that would break the Actions run.
155+
156+
### Resetting stable for every release
157+
158+
The next block is the block that fires only when I publish a new release to GitHub:
159+
160+
```yaml
161+
- name: Handle Release
162+
if: github.event_name == 'release'
163+
run: |
164+
git fetch --all
165+
git checkout stable
166+
git reset --hard ${GITHUB_REF#refs/tags/}
167+
git push origin stable --force
168+
```
169+
It does a hard reset to reset the content of the `stable` branch to the exact content of the tag that is being released, then does a force push to update the remote branch.
170+
171+
For some reason this doesn't seem to trigger that error I head earlier about the YAML workflow being updated. I'm not sure why - maybe it's because resetting to a tag is seen as a "safe" operation somehow?
172+
173+
ReadTheDocs watches for changes pushed to the `stable` branch, so this is enough to trigger a new build of the `/stable/` documentation.
174+
175+
### Copying in docs/ changes from commits marked !stable-docs
176+
177+
The last piece is the most complex. It handles that `!stable-docs` tag and, when it sees it, copies any files in `docs/` that were modified in the commit over to the `stable` branch:
178+
179+
```yaml
180+
- name: Handle Commit to Main
181+
if: contains(github.event.head_commit.message, '!stable-docs')
182+
run: |
183+
git fetch origin
184+
git checkout -b stable origin/stable
185+
# Get the list of modified files in docs/ from the current commit
186+
FILES=$(git diff-tree --no-commit-id --name-only -r ${{ github.sha }} -- docs/)
187+
# Check if the list of files is non-empty
188+
if [[ -n "$FILES" ]]; then
189+
# Checkout those files to the stable branch to over-write with their contents
190+
for FILE in $FILES; do
191+
git checkout ${{ github.sha }} -- $FILE
192+
done
193+
git add docs/
194+
git commit -m "Doc changes from ${{ github.sha }}"
195+
git push origin stable
196+
else
197+
echo "No changes to docs/ in this commit."
198+
exit 0
199+
fi
200+
```
201+
I got GPT-4 assistance with all of these (the [issue thread](https://github.com/simonw/simonw-readthedocs-experiments/issues/1) links to some of my prompts) but this was the one that took the most iteration. Let's break it down:
202+
203+
The first question we need to answer is what files in `docs/` were edited by the current commit:
204+
```bash
205+
FILES=$(git diff-tree --no-commit-id --name-only -r ${{ github.sha }} -- docs/)
206+
```
207+
If you run this command in a repo, you'll see a list of the names of the files that were modified in the most recent commit:
208+
```bash
209+
git diff-tree --no-commit-id --name-only -r main
210+
```
211+
Output is something like this:
212+
```
213+
docs/cli-reference.rst
214+
docs/cli.rst
215+
sqlite_utils/cli.py
216+
tests/test_cli.py
217+
```
218+
219+
If you add `-- docs/` at the end it will filter that down to just the files that match that path:
220+
```bash
221+
git diff-tree --no-commit-id --name-only -r main -- docs/
222+
```
223+
```
224+
docs/cli-reference.rst
225+
docs/cli.rst
226+
```
227+
The workflow assigns that to the `FILES` variable, then checks if it is empty:
228+
```bash
229+
if [[ -n "$FILES" ]]; then
230+
```
231+
If it's not empty, it loops through it and uses `git checkout` to checkout the exact copy of that file from the current commit, which will over-write the file in the current working directory:
232+
```bash
233+
for FILE in $FILES; do
234+
git checkout ${{ github.sha }} -- $FILE
235+
done
236+
```
237+
Finally, it adds those files, commits them and pushes them to the `stable` branch:
238+
```bash
239+
git add docs/
240+
git commit -m "Doc changes from ${{ github.sha }}"
241+
git push origin stable
242+
```
243+
Here's [an example commit](https://github.com/simonw/simonw-readthedocs-experiments/commit/261aeb94a07946a96e4f4b76360500db28313e43) that was created by this workflow.
244+
245+
## Configuring ReadTheDocs
246+
247+
There's one last step to putting this into action: reconfiguring ReadTheDocs to both build this new `stable` branch and to treat it as the default version for the project.
248+
249+
Here's the sequence of steps.
250+
251+
- Go to the project's admin page, click advanced settings and switch the default version from "stable" to "latest" - then scroll to the very bottom of the page to find the save button. This step is necessary because you can't delete the "stable" version if it is set as the default.
252+
- Visit the Versions tab, edit the "stable" version and uncheck the "active" box. This is the same thing as deleting it.
253+
- Find the "stable" branch on that page and activate that instead. This will trigger a build of your `stable` branch and cause it to be hosted at `/stable/` on ReadTheDocs.
254+
- Go back to the advanced settings page and switch the default version back to "stable".
255+
256+
Here's a GIF illustrating those steps:
257+
258+
![Animated GIF illustrating the sequence of steps](https://static.simonwillison.net/static/2023/readthedocs-config.gif)
259+
260+
### What this gives you
261+
262+
And that should be it! With this workflow in place and ReadTheDocs configured the following things should now be possible:
263+
264+
- Any time you ship a new release on GitHub, your `/stable/` documentation will be updated to the docs for that release.
265+
- You can fix typos in the `stable` branch and they will be quickly reflected in that `/stable/` documentation. Apply them to `main` too though or they'll be lost the next time you publish a release.
266+
- In any commit that updates a page of documentation where you want that entire page to be updated in the `/stable/` documentation, add `!stable-docs` to the commit message. The workflow will copy the full page content into the `stable` branch and trigger a new build.

0 commit comments

Comments
 (0)