Automatically sync github/etc remote repositories thorugh webhook

[agjohnson]

Currently, the remote repository list is only automatically synced when the user first connects their account. On the project import page, there is a refresh button, but this is never great UX.

Instead, we should subscribe to the necessary webhook events so that we are notified when the user's repository list changes and we can update this for the user. This conflicts slightly with our discussions around changing our relationships to RemoteRepository, but I assume we'll still want to update the mapping of User -> ??? -> RemoteRepository via webhook events.

This would avoid the page load refresh or manual refresh operation on the import page, and would keep things more consistently up to date (for webhooks subscribed to the new webhook event).

I didn't look into the other providers, but GitHub gives us a Repositories event on the webhook configuration.

Related #7169

[humitos]

It seems that subscribing to member event when adding our own webhook on GitHub should be enough to re-sync users' permissions over the repositories. I took a look at the webhook we created on our own docs project and I think this is it: https://github.com/readthedocs/readthedocs.org/settings/hooks/11220809

I tested it with test-builds by adding readthedocs-ci user and I got the webhook,

{'action': 'added',
 'changes': {'permission': {'to': 'read'}},
 'member': {
  'gravatar_id': '',
  'html_url': 'https://github.com/readthedocs-ci',
  'id': 41582472,
  'login': 'readthedocs-ci',
  'node_id': 'MDQ6VXNlcjQxNTgyNDcy',
  'site_admin': False,
  'type': 'User',
  'url': 'https://api.github.com/users/readthedocs-ci'},
 'organization': {
  'description': '',
  'id': 366329,
  'login': 'readthedocs',
  'node_id': 'MDEyOk9yZ2FuaXphdGlvbjM2NjMyOQ==',
  'url': 'https://api.github.com/orgs/readthedocs'},
 'repository': {
  'archived': False,
  'created_at': '2018-12-05T10:04:35Z',
  'default_branch': 'master',
  'description': 'Different scenarios (one per branch) to test different build configs on production',
  'disabled': False,
  'fork': False,
  'forks': 6,
  'forks_count': 6,
  'full_name': 'readthedocs/test-builds',
  'has_downloads': True,
  'has_issues': True,
  'has_pages': False,
  'has_projects': True,
  'has_wiki': True,
  'homepage': 'https://test-builds.readthedocs.io/en/latest/',
  'id': 160498353,
  'language': None,
  'license': None,
  'mirror_url': None,
  'name': 'test-builds',
  'node_id': 'MDEwOlJlcG9zaXRvcnkxNjA0OTgzNTM=',
  'open_issues': 3,
  'open_issues_count': 3,
  'owner': {
   'gravatar_id': '',
   'html_url': 'https://github.com/readthedocs',
   'id': 366329,
   'login': 'readthedocs',
   'node_id': 'MDEyOk9yZ2FuaXphdGlvbjM2NjMyOQ==',
   'type': 'Organization',
   'url': 'https://api.github.com/users/readthedocs'},
  'private': False,
  'pushed_at': '2020-07-22T13:47:02Z',
  'size': 1794,
  'ssh_url': '[email protected]:readthedocs/test-builds.git',
  'stargazers_count': 1,
  'updated_at': '2020-04-28T17:30:51Z',
  'url': 'https://api.github.com/repos/readthedocs/test-builds',
  'watchers': 1,
  'watchers_count': 1},
 'sender': {
  'gravatar_id': '',
  'html_url': 'https://github.com/humitos',
  'id': 244656,
  'login': 'humitos',
  'node_id': 'MDQ6VXNlcjI0NDY1Ng==',
  'repos_url': 'https://api.github.com/users/humitos/repos',
  'site_admin': False,
  'type': 'User',
  'url': 'https://api.github.com/users/humitos'}}

I removed all the *_url fields to make it simpler.

I think we have enough data with

• action
• changes.permission.to
• member.id
• repository.full_name

We should create a RemoteRepository like,

RemoteRepository.objects.create(
  full_name=data.repository.full_name,
  users=[User.objects.get(
    socialaccounts__provider='github',
    socialaccounts__user_id=data.member.id,
    )],
  admin=True if data.changes.permissions.to == 'admin' else False,
)

Another approach is just trigger the re-sync of all GitHub repositories for the member.id user. It's a little slower but safer to do a full re-sync for all the repositories/organizations for that user.

[stsewd]

Projects can also get out of sync if the clone URL of the project is changed to point to another project, the project in our DB will continue to be linked to the old project. We should probably restrict that field to read-only when a project isn't manually imported, and instead sync the clone URL from the repository.