Development: update initial fixture

[humitos]

We are loading test_data.json from https://github.com/readthedocs/readthedocs.org/blob/40bc5b1c01a8edfcaf4b898e5b79565598e36d51/readthedocs/projects/fixtures/test_data.json when running inv docker.up --init at https://github.com/readthedocs/common/blob/0620e5dad7831c8c6d093efb7e5c91d8118d9c61/dockerfiles/entrypoints/web.sh#L11

That fixture contains some projects that are old or are abandoned and are not good test cases to have under the development environment. It would be good to update them with better projects for testing. Those project includes test-builds, docs, sphinx-notfound-page, sphinx-hoverxref, blog and some other projects maintained by the community, like poliastro, geopandas, godot, numpy, xarray, sphinx, etc.

[tanvimoharir]

Hi,
I would like to take this up. Just to clarify, I need to add the aforementioned projects in the test_data.json file right?
Thanks

[humitos]

Yes. The idea here is to re-create the test_data.json from scratch and include useful (*) and/or well-known projects. I mentioned some of them, but we could find more as well.

(*) by useful, I mean projects with particularities: use conda, use mamba, build fast, build slow, use the config file v2, use build.tools config, etc.

[tanvimoharir]

Hi @humitos ,
I do have some doubts:

1. Could you help me understand the structure of current data which we have in test_data.json, I see three different types of json structures (pasted them below for reference)

{
        "pk": 1,
        "model": "projects.project",
        "fields": {
            "modified_date": "2010-08-15T14:36:46+00:00",
            "description": "",
            "project_url": "",
            "repo": "https://github.com/rtfd/readthedocs.org",
            "users": [
        1
        ],
            "pub_date": "2010-08-14T01:37:58+00:00",
            "slug": "read-the-docs",
            "name": "Read The Docs"
        }
{
        "pk": 1,
        "model": "builds.version",
        "fields": {
            "uploaded": false,
            "built": true,
            "project": 1,
            "active": true,
            "identifier": "2ff3d36340fa4d3d39424e8464864ca37c5f191c",
            "verbose_name": "0.2.1",
            "slug": "0.2.1"
        }
    },
    {
        "pk": 97214,
        "model": "auth.user",
        "fields": {
            "username": "test",
            "first_name": "Testy",
            "last_name": "Testerson",
            "is_active": true,
            "is_superuser": true,
            "is_staff": true,
            "last_login": "2010-08-14T01:51:05+00:00",
            "groups": [],
            "user_permissions": [],
            "password": "pbkdf2_sha256$30000$Vs87OlKZEzCb$nUw1o5pGQw7ff/QhnleSpUOupBaT1DogZrVaoZyQRyc=",
            "email": "[email protected]",
            "date_joined": "2010-08-14T01:50:58+00:00"
        }

Yes. The idea here is to re-create the test_data.json from scratch and include useful (*) and/or well-known projects. I mentioned some of them, but we could find more as well.

(*) by useful, I mean projects with particularities: use conda, use mamba, build fast, build slow, use the config file v2, use build.tools config, etc.

2. Can you also guide me a bit on how to find these projects?

Sorry if my questions are redundant, I haven't worked on this before and would like to get everything clarified.

[humitos]

Could you help me understand the structure of current data which we have in test_data.json, I see three different types of json structures (pasted them below for reference)

We use Django fixtures for this. You can read the howto guide about this from https://docs.djangoproject.com/en/3.2/howto/initial-data/

Can you also guide me a bit on how to find these projects?

We can start with the projects I mentioned already for now.

[tanvimoharir]

@humitos I added one project in test_data.json

readthedocs.org/readthedocs/projects/fixtures/test_data.json

Lines 194 to 209 in 44a9d53

	{
	"pk": 23,
	"model": "projects.project",
	"fields": {
	"modified_date": "2010-08-15T13:18:16+00:00",
	"description": "",
	"project_url": "",
	"repo": "https://github.com/sphinx-doc/sphinx",
	"users": [
	1
	],
	"pub_date": "2010-08-14T02:16:35+00:00",
	"slug": "sphinx",
	"name": "Sphinx"
	}
	},

For "pk" I added the next biggest sequential number (last was 22). However I'm not sure about "modified_date" and "pub_date" (they have the same value as one of the other projects). What values should be added for these date fields?

[humitos]

Hi @tanvimoharir! You shouldn't modify this file by hand. You should use the Django management command to export the data from your database once you have imported all the projects required. In the howto guide that I shared, this is explained step by step.

[tanvimoharir]

Hi @tanvimoharir! You shouldn't modify this file by hand. You should use the Django management command to export the data from your database once you have imported all the projects required. In the howto guide that I shared, this is explained step by step.

Thanks for explaining it to me( the how-to guide had mentioned adding fixtures manually which is why I tried that)
I have added 3 new projects directly to db and exported that into test_data.json (via dumpdata command):
https://github.com/readthedocs/readthedocs.org/pull/8948/files
If this looks okay to you then I can add other mentioned projects.
(I haven't removed any of the existing projects.)
Let me know.

[benjaoming]

Are any automated tests depending on the exact structure of these fixtures? I'm guessing I'll find out from trying to remove them now :)

[humitos]

@benjaoming no, there are not. These JSON are created automatically by Django management command. However, I proposed a different and simpler approach to follow with regular Python code instead of the usage of these fixtures. That's probably a better solution to this. See #9319 (comment)

[stsewd]

We do have tests that load the fixtures

readthedocs.org/readthedocs/rtd_tests/tests/test_repo_parsing.py

Line 7 in 3e70b81

fixtures = ['eric', 'test_data']

[benjaoming]

Thanks! I'll go through this test to ensure it's not affected by the updates of the fixtures.

Good idea @humitos to have separate fixtures for tests vs. "demo". Test fixtures are nice to maintain close to the tests or inside the tests. Will open an issue 👍

[benjaoming]

Keeping this issue open, as it seems like the cherry-picked commit 546ada4 in #9319 did not include the requested updates, it was instead a merely expanding existing data. Will need to redo this, but preferring the method outlined in #9332