Remove old symlink when Domain is changed

[humitos]

I started getting random 404 at https://docs.pytest.org/en/latest/ and we found they have confifured doc.pytest.org as a their domain so, the new web server was failing on serving the files.

At some point, they should have configured docs.pytest.org under RTD and then they changed it to just doc, otherwise RTD should know nothing about the docs domain in the old web servers.

To fix this, once the Domain is changed we need to perform a cleanup of the old symlinks in the web servers. This can be done by a task that catches all the various symlink leftovers we have, and sync’s them.

• https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/views/private.py#L678-L701
• https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/forms.py#L546

As a reference, they are mixing doc and docs in their repo: https://github.com/pytest-dev/pytest/blob/0a15edd5732578f6cc40e28770f0fc87dfa5ced9/doc/en/announce/release-3.2.0.rst

[ericholscher]

We should do two things:

• Change the logic of the Domain Form or View to trigger a delete symlink task
• Have a background task that runs every hour that checks for invalid symlinks, then deletes them if found

The change should be immediate when users act, but it's nice to have something in the background doing cleanup. It also catches a number of cases the Views & Forms don't catch (eg. if we edit it in the Admin, CLI, ORM update() etc.).

[ericholscher]

It looks like our symlink domain logic already accounts for deletions, we just aren't calling it: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/tasks.py#L775-L801

[humitos]

I'm not really sure how the Domain works under RTD, but I see there is a count attribute (Number of times this domain has been hit) in the model that won't have more sense if we modify/edit the domain object.

Taking a look at the code, it doesn't seem to be tought to allow the edit operation. It looks like more delete and create a new one. In this case, the delete method will remove the old symlinks properly since it's calling the proper task.

https://github.com/rtfd/readthedocs.org/blob/1af444173481df9a7b16e015fc6b12abe8155e7e/readthedocs/projects/models.py#L1014-L1018

Regarding the background task, what's the proper way to do this? How do we search for orphan domains?

1. list all the symlinks under (Public/Private)Symlink.CNAME_ROOT
2. use each of them to query the database and check the Domain object exists
3. if it doesn't exist, remove the symlink

Sounds good?

[humitos]

To modify the Domain Form or view, we will need to modify the symlink_domain task to accept a domain_name instead of a domain_pk since the domain name that we want to remove doesn't exist anymore in the database but in the filesystem. So, at the moment we don't have a way to remove a "domain name" that it's not a "domain object" in the database.

I see two possible solutions:

1. do not allow edit at all and force the user to delete and create a new one
2. broadcast the periodic task immediately when the domain is modified (kind of overkill)

I think that 1) makes more sense after reading the code.