Remote Repository and Remote Organization Normalization

[saadmk11]

This PR contains the full implementation for the Remote Repository and Remote Organization Normalization.

This PR creates separate database tables for the RemoteRepository and RemoteOrganization with the new modeling.
This also updates the codebase with the changes for the new modeling and adds a management command to re-sync all the remote repositories and remoter organizations.

The tasks done in this PR can be found here: https://github.com/orgs/readthedocs/projects/74#column-11164063

Closes #7572
Closes #7727

[humitos]

I did some research and tests for the database constraint that we will need to deal with. The plan here will be:

1. remove all the constraints from the new tables (oauth_remoterepository_2020, oauth_remoterepository_relations, oauth_remoteorganization_relations) that reference to current used tables (socialaccount_socialaccount, auth_user, etc) that could be deleted while doing the 1-week resync
2. do the resync during that week
3. re-add these constraints as NOT VALID (more below) so deleted users/accounts are not validated, but all new data is validated before insert/update
4. remove all rows from new tables that are not valid (the row referenced in the other table was deleted) --this step is not strict, but is a good cleanup
5. run ALTER TABLE ... VALIDATE CONSTRAINT ... after the table is cleanup from invalid rows

PostgreSQL documentation: https://www.postgresql.org/docs/9.1/sql-altertable.html

ADD table_constraint [ NOT VALID ]
This form adds a new constraint to a table using the same syntax as CREATE TABLE, plus the option NOT VALID, which is currently only allowed for foreign key constraints. If the constraint is marked NOT VALID, the potentially-lengthy initial check to verify that all rows in the table satisfy the constraint is skipped. The constraint will still be enforced against subsequent inserts or updates (that is, they'll fail unless there is a matching row in the referenced table). But the database will not assume that the constraint holds for all rows in the table, until it is validated by using the VALIDATE CONSTRAINT option.

As example in my local instance:

• DROP CONSTRAINT

# Drop constraints from RemoteRepositoryRelation
ALTER TABLE oauth_remoterepositoryrelation DROP CONSTRAINT oauth_remotereposito_account_id_02d56944_fk_socialacc;
ALTER TABLE oauth_remoterepositoryrelation DROP CONSTRAINT oauth_remoterepositoryrelation_user_id_3c2baf85_fk_auth_user_id;

# Drop constraints from RemoteOrganizationRelation
ALTER TABLE oauth_remoteorganizationrelation DROP CONSTRAINT oauth_remoteorganiza_account_id_8dbef288_fk_socialacc;
ALTER TABLE oauth_remoteorganizationrelation DROP CONSTRAINT oauth_remoteorganiza_user_id_0603492b_fk_auth_user;

• ADD CONSTRAINT

# Add constraints from RemoteRepositoryRelation
ALTER TABLE "oauth_remoterepositoryrelation" ADD CONSTRAINT "oauth_remoterepositoryrelation_user_id_3c2baf85_fk_auth_user_id" FOREIGN KEY ("user_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED;

ALTER TABLE "oauth_remoterepositoryrelation" ADD CONSTRAINT "oauth_remotereposito_account_id_02d56944_fk_socialacc" FOREIGN KEY ("account_id") REFERENCES "socialaccount_socialaccount" ("id") DEFERRABLE INITIALLY DEFERRED;


# Add constraints for RemoteOrganizationRelation
ALTER TABLE "oauth_remoteorganizationrelation" ADD CONSTRAINT "oauth_remoteorganiza_user_id_0603492b_fk_auth_user" FOREIGN KEY ("user_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED;

ALTER TABLE "oauth_remoteorganizationrelation" ADD CONSTRAINT "oauth_remoteorganiza_account_id_8dbef288_fk_socialacc" FOREIGN KEY ("account_id") REFERENCES "socialaccount_socialaccount" ("id") DEFERRABLE INITIALLY DEFERRED;

First, try to ADD CONSTRAINT as they are here (changing the name, probably for the production ones) and if they fail saying that there are some invalid data, add NOT VALID at the end. After that, do the cleanup step.

Note I found the ADD CONSTRAINT commands by running inv docker.manage sqlmigrate oauth 0013 (and 0012)

[ericholscher]

NOT VALID is awesome! This sounds like a good plan 👍

[ericholscher]

Changes look good!

[humitos]

Just to mention that I linked docs from 9.1 and we are using v12. The documentation is a updated and better: https://www.postgresql.org/docs/12/sql-altertable.html

First, try to ADD CONSTRAINT as they are here (changing the name, probably for the production ones) and if they fail saying that there are some invalid data, add NOT VALID at the end. After that, do the cleanup step.

After reading the v12 docs it seems it's better to use NOT VALID at first (to avoid a huge scan on the table that will lock it) and once it's done use VALIDATE CONSTRAINT that won't lock the table. However, from the Notes section,

validation acquires only a SHARE UPDATE EXCLUSIVE lock on the table being altered. (If the constraint is a foreign key then a ROW SHARE lock is also required on the table referenced by the constraint.)

it seems we may need to lock the other tables? (Project and SocialAcctount)

[humitos]

This PR is merged in rel and already deployed in Community. I'm not merging it into master yet because we don't want to be deployed in Commercial yet. We are still doing some QA on .org and we haven't triggered the re-sync task on .com yet.

[humitos]

This PR is ready to be merged and it will go out tomorrow in commercial.