Skip to content

Commit 501ecfa

Browse files
stsewdstsewd
authored
Use django-safemigrate for migrations (#11087)
* Use django-safemigrate for migrations Closes #10964 * Add to toc * Fix migration * Fix fixtures * Update common * Fix package name * Updates from review * Update common
1 parent e674df0 commit 501ecfa

File tree

16 files changed

+307
-64
lines changed

16 files changed

+307
-64
lines changed

‎docs/dev/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ or taking the open source Read the Docs codebase for your own custom installatio
2525
style-guide
2626
front-end
2727
i18n
28+
migrations
2829
server-side-search
2930
search-integration
3031
subscriptions

‎docs/dev/migrations.rst

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
Database migrations
2+
===================
3+
4+
We use `Django migrations <https://docs.djangoproject.com/en/4.2/topics/migrations/>`__ to manage database schema changes,
5+
and the `django-safemigrate <https://github.com/aspiredu/django-safemigrate>`__ package to ensure that migrations are run in a given order to avoid downtime.
6+
7+
To make sure that migrations don't cause downtime,
8+
the following rules should be followed for each case.
9+
10+
Adding a new field
11+
------------------
12+
13+
**When adding a new field to a model, it should be nullable.**
14+
This way, the database can be migrated without downtime, and the field can be populated later.
15+
Don't forget to make the field non-nullable in a separate migration after the data has been populated.
16+
You can achieve this by following these steps:
17+
18+
- #. Set the new field as ``null=True`` and ``blank=True`` in the model.
19+
20+
.. code-block:: python
21+
22+
class MyModel(models.Model):
23+
new_field = models.CharField(
24+
max_length=100, null=True, blank=True, default="default"
25+
)
26+
27+
- #. Make sure that the field is always populated with a proper value in the new code,
28+
and the code handles the case where the field is null.
29+
30+
.. code-block:: python
31+
32+
if my_model.new_field in [None, "default"]:
33+
pass
34+
35+
36+
# If it's a boolean field, make sure that the null option is removed from the form.
37+
class MyModelForm(forms.ModelForm):
38+
def __init__(self, *args, **kwargs):
39+
super().__init__(*args, **kwargs)
40+
self.fields["new_field"].widget = forms.CheckboxInput()
41+
self.fields["new_field"].empty_value = False
42+
43+
- #. Create the migration file (let's call this migration ``app 0001``),
44+
and mark it as ``Safe.before_deploy``.
45+
46+
.. code-block:: python
47+
48+
from django.db import migrations, models
49+
from django_safemigrate import Safe
50+
51+
52+
class Migration(migrations.Migration):
53+
safe = Safe.before_deploy
54+
55+
- #. Create a data migration to populate all null values of the new field with a proper value (let's call this migration ``app 0002``),
56+
and mark it as ``Safe.after_deploy``.
57+
58+
.. code-block:: python
59+
60+
from django.db import migrations
61+
62+
63+
def migrate(apps, schema_editor):
64+
MyModel = apps.get_model("app", "MyModel")
65+
MyModel.objects.filter(new_field=None).update(new_field="default")
66+
67+
68+
class Migration(migrations.Migration):
69+
safe = Safe.after_deploy
70+
71+
operations = [
72+
migrations.RunPython(migrate),
73+
]
74+
75+
- #. After the deploy has been completed, create a new migration to set the field as non-nullable (let's call this migration ``app 0003``).
76+
Run this migration on a new deploy, you can mark it as ``Safe.before_deploy`` or ``Safe.always``.
77+
- #. Remove any handling of the null case from the code.
78+
79+
At the end, the deploy should look like this:
80+
81+
- Deploy web-extra.
82+
- Run ``django-admin safemigrate`` to run the migration ``app 0001``.
83+
- Deploy the webs
84+
- Run ``django-admin migrate`` to run the migration ``app 0002``.
85+
- Create a new migration to set the field as non-nullable,
86+
and apply it on the next deploy.
87+
88+
Removing a field
89+
----------------
90+
91+
**When removing a field from a model,
92+
all usages of the field should be removed from the code before the field is removed from the model,
93+
and the field should be nullable.**
94+
You can achieve this by following these steps:
95+
96+
- #. Remove all usages of the field from the code.
97+
- #. Set the field as ``null=True`` and ``blank=True`` in the model.
98+
99+
.. code-block:: python
100+
101+
class MyModel(models.Model):
102+
field_to_delete = models.CharField(max_length=100, null=True, blank=True)
103+
104+
- #. Create the migration file (let's call this migration ``app 0001``),
105+
and mark it as ``Safe.before_deploy``.
106+
107+
.. code-block:: python
108+
109+
from django.db import migrations, models
110+
from django_safemigrate import Safe
111+
112+
113+
class Migration(migrations.Migration):
114+
safe = Safe.before_deploy
115+
116+
- #. Create a migration to remove the field from the database (let's call this migration ``app 0002``),
117+
and mark it as ``Safe.after_deploy``.
118+
119+
.. code-block:: python
120+
121+
from django.db import migrations, models
122+
from django_safemigrate import Safe
123+
124+
125+
class Migration(migrations.Migration):
126+
safe = Safe.after_deploy
127+
128+
At the end, the deploy should look like this:
129+
130+
- Deploy web-extra.
131+
- Run ``django-admin safemigrate`` to run the migration ``app 0001``.
132+
- Deploy the webs
133+
- Run ``django-admin migrate`` to run the migration ``app 0002``.

‎readthedocs/builds/migrations/0057_migrate_timestamp_fields.py

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Generated by Django 4.2.9 on 2024-02-01 20:29
2+
3+
import datetime
4+
5+
from django.db import migrations
6+
from django_safemigrate import Safe
7+
8+
9+
def migrate(apps, schema_editor):
10+
"""
11+
Migrate the created and modified fields of the Version model to have a non-null value.
12+
13+
This date corresponds to the release date of 5.6.5,
14+
when the created field was added to the Version model
15+
at https://github.com/readthedocs/readthedocs.org/commit/d72ee6e27dc398b97e884ccec8a8cf135134faac.
16+
"""
17+
Version = apps.get_model("builds", "Version")
18+
date = datetime.datetime(2020, 11, 23, tzinfo=datetime.timezone.utc)
19+
Version.objects.filter(created=None).update(created=date)
20+
Version.objects.filter(modified=None).update(modified=date)
21+
22+
23+
class Migration(migrations.Migration):
24+
safe = Safe.before_deploy
25+
26+
dependencies = [
27+
("builds", "0056_alter_versionautomationrule_priority"),
28+
]
29+
30+
operations = [
31+
migrations.RunPython(migrate),
32+
]

‎readthedocs/builds/migrations/0058_alter_version_created_alter_version_modified.py

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Generated by Django 4.2.9 on 2024-02-01 20:38
2+
3+
import django.utils.timezone
4+
import django_extensions.db.fields
5+
from django.db import migrations
6+
from django_safemigrate import Safe
7+
8+
9+
class Migration(migrations.Migration):
10+
safe = Safe.after_deploy
11+
12+
dependencies = [
13+
("builds", "0057_migrate_timestamp_fields"),
14+
]
15+
16+
operations = [
17+
migrations.AlterField(
18+
model_name="version",
19+
name="created",
20+
field=django_extensions.db.fields.CreationDateTimeField(
21+
auto_now_add=True,
22+
default=django.utils.timezone.now,
23+
verbose_name="created",
24+
),
25+
preserve_default=False,
26+
),
27+
migrations.AlterField(
28+
model_name="version",
29+
name="modified",
30+
field=django_extensions.db.fields.ModificationDateTimeField(
31+
auto_now=True,
32+
default=django.utils.timezone.now,
33+
verbose_name="modified",
34+
),
35+
preserve_default=False,
36+
),
37+
]

‎readthedocs/builds/models.py

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@
1313
from django.utils import timezone
1414
from django.utils.translation import gettext
1515
from django.utils.translation import gettext_lazy as _
16-
from django_extensions.db.fields import CreationDateTimeField, ModificationDateTimeField
1716
from django_extensions.db.models import TimeStampedModel
1817
from polymorphic.models import PolymorphicModel
1918

@@ -92,19 +91,6 @@ class Version(TimeStampedModel):
9291

9392
"""Version of a ``Project``."""
9493

95-
# Overridden from TimeStampedModel just to allow null values.
96-
# TODO: remove after deploy.
97-
created = CreationDateTimeField(
98-
_('created'),
99-
null=True,
100-
blank=True,
101-
)
102-
modified = ModificationDateTimeField(
103-
_('modified'),
104-
null=True,
105-
blank=True,
106-
)
107-
10894
project = models.ForeignKey(
10995
Project,
11096
verbose_name=_('Project'),

‎readthedocs/integrations/migrations/0013_set_timestamp_fields_as_no_null.py

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Generated by Django 4.2.9 on 2024-02-01 20:10
2+
3+
import django.utils.timezone
4+
import django_extensions.db.fields
5+
from django.db import migrations
6+
from django_safemigrate import Safe
7+
8+
9+
class Migration(migrations.Migration):
10+
safe = Safe.always
11+
12+
dependencies = [
13+
("integrations", "0012_migrate_timestamp_fields"),
14+
]
15+
16+
operations = [
17+
migrations.AlterField(
18+
model_name="integration",
19+
name="created",
20+
field=django_extensions.db.fields.CreationDateTimeField(
21+
auto_now_add=True,
22+
default=django.utils.timezone.now,
23+
verbose_name="created",
24+
),
25+
preserve_default=False,
26+
),
27+
migrations.AlterField(
28+
model_name="integration",
29+
name="modified",
30+
field=django_extensions.db.fields.ModificationDateTimeField(
31+
auto_now=True,
32+
default=django.utils.timezone.now,
33+
verbose_name="modified",
34+
),
35+
preserve_default=False,
36+
),
37+
]

‎readthedocs/integrations/models.py

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
from django.utils.crypto import get_random_string
1010
from django.utils.safestring import mark_safe
1111
from django.utils.translation import gettext_lazy as _
12-
from django_extensions.db.fields import CreationDateTimeField, ModificationDateTimeField
1312
from django_extensions.db.models import TimeStampedModel
1413
from pygments import highlight
1514
from pygments.formatters import HtmlFormatter
@@ -278,19 +277,6 @@ class Integration(TimeStampedModel):
278277

279278
INTEGRATIONS = WEBHOOK_INTEGRATIONS
280279

281-
# Overridden from TimeStampedModel just to allow null values.
282-
# TODO: remove after deploy.
283-
created = CreationDateTimeField(
284-
_("created"),
285-
null=True,
286-
blank=True,
287-
)
288-
modified = ModificationDateTimeField(
289-
_("modified"),
290-
null=True,
291-
blank=True,
292-
)
293-
294280
project = models.ForeignKey(
295281
Project,
296282
related_name="integrations",

‎readthedocs/projects/fixtures/test_data.json

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -888,8 +888,8 @@
888888
"model": "builds.version",
889889
"pk": 1,
890890
"fields": {
891-
"created": null,
892-
"modified": null,
891+
"created": "2022-02-22T10:55:46.784Z",
892+
"modified": "2022-02-22T10:55:46.784Z",
893893
"project": 1,
894894
"type": "unknown",
895895
"identifier": "2ff3d36340fa4d3d39424e8464864ca37c5f191c",
@@ -912,8 +912,8 @@
912912
"model": "builds.version",
913913
"pk": 2,
914914
"fields": {
915-
"created": null,
916-
"modified": null,
915+
"created": "2022-02-22T10:55:46.784Z",
916+
"modified": "2022-02-22T10:55:46.784Z",
917917
"project": 1,
918918
"type": "unknown",
919919
"identifier": "354456a7dba2a75888e2fe91f6d921e5fe492bcd",
@@ -936,8 +936,8 @@
936936
"model": "builds.version",
937937
"pk": 3,
938938
"fields": {
939-
"created": null,
940-
"modified": null,
939+
"created": "2022-02-22T10:55:46.784Z",
940+
"modified": "2022-02-22T10:55:46.784Z",
941941
"project": 1,
942942
"type": "unknown",
943943
"identifier": "master",
@@ -960,8 +960,8 @@
960960
"model": "builds.version",
961961
"pk": 4,
962962
"fields": {
963-
"created": null,
964-
"modified": null,
963+
"created": "2022-02-22T10:55:46.784Z",
964+
"modified": "2022-02-22T10:55:46.784Z",
965965
"project": 1,
966966
"type": "unknown",
967967
"identifier": "not_ok",
@@ -984,8 +984,8 @@
984984
"model": "builds.version",
985985
"pk": 8,
986986
"fields": {
987-
"created": null,
988-
"modified": null,
987+
"created": "2022-02-22T10:55:46.784Z",
988+
"modified": "2022-02-22T10:55:46.784Z",
989989
"project": 1,
990990
"type": "unknown",
991991
"identifier": "awesome",
@@ -1008,8 +1008,8 @@
10081008
"model": "builds.version",
10091009
"pk": 18,
10101010
"fields": {
1011-
"created": null,
1012-
"modified": null,
1011+
"created": "2022-02-22T10:55:46.784Z",
1012+
"modified": "2022-02-22T10:55:46.784Z",
10131013
"project": 6,
10141014
"type": "tag",
10151015
"identifier": "2404a34eba4ee9c48cc8bc4055b99a48354f4950",
@@ -1032,8 +1032,8 @@
10321032
"model": "builds.version",
10331033
"pk": 19,
10341034
"fields": {
1035-
"created": null,
1036-
"modified": null,
1035+
"created": "2022-02-22T10:55:46.784Z",
1036+
"modified": "2022-02-22T10:55:46.784Z",
10371037
"project": 6,
10381038
"type": "tag",
10391039
"identifier": "f55c28e560c92cafb6e6451f8084232b6d717603",

0 commit comments

Comments
 (0)