Add GitHub App service (#12072)

This is the big one :D

- After this PR is merged, we will start keeping in sync installations
and remote repositories, but they won't be exposed to users yet (we can
manually set up our projects to use a remote repo to test things out,
private repos are not supported with this PR).
- All operations from the old integration are supported, except for
cloning private repos (will open another PR for that).
- PyGitHub is an okay solution, it abstracts dealing with the JWT and
has types for everything. But it can also be inefficient in some cases
(like to interact with a commit, you need to retrieve the commit from
the API first), but I think that's fine for now, we can always fallback
to do this manually, and hopefully shouldn't be that much work, or the
other option is keep using PyGitHub and use a raw call
(requester.getJsonResponse("url")).
- Webhooks always recover nicely in case the installation wasn't
tracked, so if one operation fails for whatever reason (a bug, github is
down, our app is down), we can recover the state in the next operation.
Users can also manually hit "sync".
- The webhook is exposed from the `/webhook/githubapp/` URL, we can
change that if needed.
- We need to add the secrets and IDs on the ops repos.
- Could I have divided this into smaller PRs? Maybe, but that may
require more time removing the tests that are not needed...

Extracted from
#11942

Author: stsewd

docs/dev/github-app.rst

@@ -0,0 +1,46 @@
+GitHub App
+==========
+
+Our GitHub App integration consists of a GitHub App (one for each platform, readthedocs.org and readthedocs.com),
+which can be installed on a user's account or organization.
+
+After installing the GitHub App, users can grant acccess to all repositories or select specific repositories,
+this allows Read the Docs to access the repositories and perform actions on them, such as reporting build statuses,
+and subscribe to events like push and pull request events.
+
+Unlike our other Git integrations, this doesn't rely on user's OAuth2 tokens to interact with the repositories, instead it makes use of the installation.
+As long as the installation is active and has access to the repositories, Read the Docs can perform actions on them.
+
+Installations can be `suspended <https://docs.github.com/en/apps/maintaining-github-apps/suspending-a-github-app-installation>`__,
+which means that the GitHub App is still installed but cannot access any resources, this action is reversible.
+Installations can also be uninstalled, which means that the GitHub App is removed from the account or organization,
+and cannot access any resources, this action is irreversible.
+
+You can read more about GitHub Apps in `GitHub's documentation <https://docs.github.com/en/apps/overview>`__.
+
+Modeling
+--------
+
+Since we no longer make use of user's OAuth2 tokens, we need to keep track of all the installations of our GitHub App in the database (``GitHubAppInstallation`` model).
+Each remote repository has a foreign key to the installation that has access to it,
+so we can use that installation to perform actions on the repository.
+
+Keeping data in sync
+--------------------
+
+Once a GitHub app is installed, GitHub will send webhook events to our application (``readthedocs.oauth.views``) related to the installation
+(even when the installation is uninstalled or revoked), repositories, and organizations.
+This allows us to always have the ``RemoteRepository`` and ``RemoteOrganization`` models and its permissions in sync with GitHub.
+
+The only use case for using the user's OAuth2 token is to get all the installations the user has access to,
+this helps us to keep permissions in sync with GitHub.
+
+Security
+--------
+
+- All webhooks are signed with a secret, which we verify before processing each event.
+- Since we make use of the installation to perform actions on the repositories instead of the user's OAuth2 token,
+  we make sure that only users with admin permissions on the repository can link the repository to a Read the Docs project.
+- Once we lose access to a repository (e.g. the installation is uninstalled or revoked, or the project was deselected from the installation),
+  we remove the remote repository from the database, as we don't want to keep the relation bettween the project and the repository.
+  This is to prevent connecting the repository to the project again without the user's consent if they grant access to the repository again.

docs/dev/index.rst

@@ -29,5 +29,6 @@ or taking the open source Read the Docs codebase for your own custom installatio
    server-side-search
    search-integration
    subscriptions
+   github-app
    settings
    tests

docs/dev/install.rst

@@ -262,10 +262,32 @@ For others, the webhook will simply fail to connect when there are new commits t
 
 * Configure the applications on GitHub, Bitbucket, and GitLab.
   For each of these, the callback URI is ``http://devthedocs.org/accounts/<provider>/login/callback/``
-  where ``<provider>`` is one of ``github``, ``gitlab``, or ``bitbucket_oauth2``.
+  where ``<provider>`` is one of ``github``, ``githubapp``, ``gitlab``, or ``bitbucket_oauth2``.
   When setup, you will be given a "Client ID" (also called an "Application ID" or just "Key") and a "Secret".
 * Take the "Client ID" and "Secret" for each service and set them as :ref:`environment variables <settings:Allauth secrets>`.
 
+Configuring GitHub App
+~~~~~~~~~~~~~~~~~~~~~~
+
+- Create a new GitHub app from https://github.com/settings/apps/new.
+- Callback URL should be ``http://devthedocs.org/accounts/githubapp/login/callback/``.
+- Keep marked "Expire user authorization tokens"
+- Activate the webhook, and set the URL to one provided by a service like `Webhook.site <https://docs.webhook.site/cli.html>`__ to forward all incoming webhooks to your local development instance.
+  You should forward all events to ``http://devthedocs.org/webhook/githubapp/``.
+- In permissions, select the following:
+
+  - Repository permissions: Commit statuses (read and write, so we can create commit statuses),
+    Contents (read only, so we can clone repos with a token),
+    Metadata (read only, so we read the repo collaborators),
+    Pull requests (read and write, so we can post a comment on PRs in the future).
+  - Organization permissions: Members (read only so we can read the organization members).
+  - Account permissions: Email addresses (read only, so allauth can fetch all verified emails).
+
+- Subscribe to the following events: Installation target, Member, Organization, Membership, Pull request, Push, and Repository.
+- Copy the "Client ID" and "Client Secret" and set them as :ref:`environment variables <settings:Allauth secrets>`.
+- Generate a webhook secret and a private key from the GitHub App settings,
+  and set them as :ref:`environment variables <settings:GitHub App secrets>`.
+
 Troubleshooting
 ---------------
 

docs/dev/settings.rst

@@ -167,6 +167,16 @@ providers using the following environment variables:
 .. envvar:: RTD_SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID
 .. envvar:: RTD_SOCIALACCOUNT_PROVIDERS_GOOGLE_SECRET
 
+GitHub App
+~~~~~~~~~~
+
+You can use the following environment variables to set the settings used by the GitHub App:
+
+.. envvar:: RTD_GITHUB_APP_ID
+.. envvar:: RTD_GITHUB_APP_NAME
+.. envvar:: RTD_GITHUB_APP_PRIVATE_KEY
+.. envvar:: RTD_GITHUB_APP_WEBHOOK_SECRET
+
 Stripe secrets
 ~~~~~~~~~~~~~~
 

readthedocs/oauth/clients.py

@@ -1,7 +1,10 @@
 from datetime import datetime
 
 import structlog
+from django.conf import settings
 from django.utils import timezone
+from github import Auth
+from github import GithubIntegration
 from requests_oauthlib import OAuth2Session
 
 
@@ -70,3 +73,15 @@ def get_oauth2_client(account):
         token_updater=_get_token_updater(token),
     )
     return session
+
+
+def get_gh_app_client() -> GithubIntegration:
+    """Return a client authenticated as the GitHub App to interact with the API."""
+    app_auth = Auth.AppAuth(
+        app_id=settings.GITHUB_APP_CLIENT_ID,
+        private_key=settings.GITHUB_APP_PRIVATE_KEY,
+        # 10 minutes is the maximum allowed by GitHub.
+        # PyGithub will handle the token expiration and renew it automatically.
+        jwt_expiry=60 * 10,
+    )
+    return GithubIntegration(auth=app_auth)

readthedocs/oauth/models.py

@@ -1,5 +1,7 @@
 """OAuth service models."""
 
+from functools import cached_property
+
 import structlog
 from allauth.socialaccount.models import SocialAccount
 from django.contrib.auth.models import User
@@ -12,6 +14,7 @@
 from readthedocs.projects.constants import REPO_CHOICES
 from readthedocs.projects.models import Project
 
+from .constants import GITHUB_APP
 from .constants import VCS_PROVIDER_CHOICES
 from .querysets import RemoteOrganizationQuerySet
 from .querysets import RemoteRepositoryQuerySet
@@ -84,6 +87,86 @@ class GitHubAppInstallation(TimeStampedModel):
     class Meta(TimeStampedModel.Meta):
         verbose_name = _("GitHub app installation")
 
+    @cached_property
+    def service(self):
+        """Return the service for this installation."""
+        from readthedocs.oauth.services import GitHubAppService
+
+        return GitHubAppService(self)
+
+    def delete(self, *args, **kwargs):
+        """Override delete method to remove orphaned remote organizations."""
+        self.delete_repositories()
+        return super().delete(*args, **kwargs)
+
+    def delete_repositories(self, repository_ids: list[int] | None = None):
+        """
+        Delete repositories linked to this installation.
+        When an installation is deleted, we delete all its remote repositories
+        and relations, users will need to manually link the projects to each repository again.
+        We also remove organizations that don't have any repositories after removing the repositories.
+        :param repository_ids: List of repository ids (remote ID) to delete.
+         If None, all repositories will be considered for deletion.
+        """
+        # repository_ids is optional (None, which means all repositories),
+        # but if it's an empty list, we don't want to delete anything.
+        if repository_ids is not None and not repository_ids:
+            log.info("No remote repositories to delete")
+            return
+
+        remote_organizations = RemoteOrganization.objects.filter(
+            repositories__github_app_installation=self,
+            vcs_provider=GITHUB_APP,
+        )
+        remote_repositories = self.repositories.filter(vcs_provider=GITHUB_APP)
+        if repository_ids:
+            remote_organizations = remote_organizations.filter(
+                repositories__remote_id__in=repository_ids
+            )
+            remote_repositories = remote_repositories.filter(remote_id__in=repository_ids)
+
+        # Fetch all IDs before deleting the repositories, so we can filter the organizations later.
+        remote_organizations_ids = list(remote_organizations.values_list("id", flat=True))
+
+        count, deleted = remote_repositories.delete()
+        log.info(
+            "Deleted remote repositories",
+            count=count,
+            deleted=deleted,
+            installation_id=self.installation_id,
+            target_id=self.target_id,
+            target_type=self.target_type,
+        )
+
+        count, deleted = RemoteOrganization.objects.filter(
+            id__in=remote_organizations_ids,
+            repositories=None,
+        ).delete()
+        log.info(
+            "Deleted orphaned remote organizations",
+            count=count,
+            deleted=deleted,
+            installation_id=self.installation_id,
+            target_id=self.target_id,
+            target_type=self.target_type,
+        )
+
+    def delete_organization(self, organization_id: int):
+        """Delete a remote organization and all its remote repositories and relations from the database."""
+        count, deleted = RemoteOrganization.objects.filter(
+            remote_id=str(organization_id),
+            vcs_provider=GITHUB_APP,
+        ).delete()
+        log.info(
+            "Deleted remote organization",
+            count=count,
+            deleted=deleted,
+            organization_id=organization_id,
+            installation_id=self.installation_id,
+            target_id=self.target_id,
+            target_type=self.target_type,
+        )
+
 
 class RemoteOrganization(TimeStampedModel):
     """

readthedocs/oauth/querysets.py

@@ -3,6 +3,7 @@
 from django.db import models
 
 from readthedocs.core.querysets import NoReprQuerySet
+from readthedocs.oauth.constants import GITHUB_APP
 
 
 class RelatedUserQuerySet(NoReprQuerySet, models.QuerySet):
@@ -12,7 +13,11 @@ def api(self, user=None):
         """Return objects for user."""
         if not user.is_authenticated:
             return self.none()
-        return self.filter(users=user)
+        queryset = self.filter(users=user)
+        # TODO: Once we are migrated into GitHub App we should include these repositories/organizations.
+        # Exclude repositories/organizations from the GitHub App for now to avoid duplicated entries.
+        queryset = queryset.exclude(vcs_provider=GITHUB_APP)
+        return queryset
 
     def api_v2(self, *args, **kwargs):
         # API v2 is the same as API v3 for .org, but it's
@@ -28,10 +33,14 @@ def for_project_linking(self, user):
         Repositories can be linked to a project only if the user has admin access
         to the repository on the VCS service.
         """
-        return self.filter(
+        queryset = self.filter(
             remote_repository_relations__user=user,
             remote_repository_relations__admin=True,
-        ).distinct()
+        )
+        # TODO: Once we are migrated into GitHub App we should include these repositories/organizations.
+        # Exclude repositories/organizations from the GitHub App for now to avoid duplicated entries.
+        queryset = queryset.exclude(vcs_provider=GITHUB_APP)
+        return queryset.distinct()
 
 
 class RemoteOrganizationQuerySet(RelatedUserQuerySet):

readthedocs/oauth/services/__init__.py

@@ -4,6 +4,16 @@
 from readthedocs.oauth.services import bitbucket
 from readthedocs.oauth.services import github
 from readthedocs.oauth.services import gitlab
+from readthedocs.oauth.services.githubapp import GitHubAppService
+
+
+__all__ = [
+    "GitHubService",
+    "BitbucketService",
+    "GitLabService",
+    "GitHubAppService",
+    "registry",
+]
 
 
 class GitHubService(SettingsOverrideObject):
@@ -21,4 +31,4 @@ class GitLabService(SettingsOverrideObject):
     _override_setting = "OAUTH_GITLAB_SERVICE"
 
 
-registry = [GitHubService, BitbucketService, GitLabService]
+registry = [GitHubService, BitbucketService, GitLabService, GitHubAppService]

readthedocs/oauth/services/base.py

@@ -34,7 +34,7 @@ class Service:
     vcs_provider_slug: str
     allauth_provider = type[OAuth2Provider]
 
-    url_pattern: re.Pattern | None
+    url_pattern: re.Pattern | None = None
     default_user_avatar_url = settings.OAUTH_AVATAR_USER_DEFAULT_URL
     default_org_avatar_url = settings.OAUTH_AVATAR_ORG_DEFAULT_URL
     supports_build_status = False
@@ -106,6 +106,10 @@ def send_build_status(self, *, build, commit, status):
         """
         raise NotImplementedError
 
+    def get_clone_token(self, project):
+        """Get a token used for cloning the repository."""
+        raise NotImplementedError
+
     @classmethod
     def is_project_service(cls, project):
         """
@@ -326,3 +330,7 @@ def sync_repositories(self):
 
     def sync_organizations(self):
         raise NotImplementedError
+
+    def get_clone_token(self, project):
+        """User services make use of SSH keys only for cloning."""
+        return None