diff --git a/docs/user/guides/connecting-git-account.rst b/docs/user/guides/connecting-git-account.rst index 9410ffe2929..c2a5648529f 100644 --- a/docs/user/guides/connecting-git-account.rst +++ b/docs/user/guides/connecting-git-account.rst @@ -40,7 +40,7 @@ You will now see the account appear in the list of connected services. :align: center :alt: Screenshot of Read the Docs "Connected Services" page with multiple services connected - Connected Services [#f1]_ [#f2]_ shows the list of Git providers that + Connected Services [#f1]_ [#f2]_ shows the list of Git providers that are connected to your Read the Docs account. Now your connection is ready and you will be able to import and configure Git repositories with just a few clicks. @@ -57,11 +57,11 @@ You may at any time delete the connection from Read the Docs. Delete the connection makes Read the Docs forget the immediate access, but you should also disable our OAuth Application from your Git provider. -* On GitHub, navigate to `Authorized OAuth Apps`_. +* On GitHub, navigate to `Authorized GitHub Apps`_. * On Bitbucket, navigate to `Application Authorizations`_. * On GitLab, navigat to `Applications`_ -.. _Authorized OAuth Apps: https://github.com/settings/applications +.. _Authorized GitHub Apps: https://github.com/settings/apps/authorizations .. _Application Authorizations: https://bitbucket.org/account/settings/app-authorizations/ .. _Applications: https://gitlab.com/-/profile/applications diff --git a/docs/user/guides/private-submodules.rst b/docs/user/guides/private-submodules.rst index 0bc67d69310..eb9ea167a4f 100644 --- a/docs/user/guides/private-submodules.rst +++ b/docs/user/guides/private-submodules.rst @@ -8,9 +8,12 @@ How to use private Git submodules If you are using private Git repositories and they also contain private Git submodules, you need to follow a few special steps. -Read the Docs uses SSH keys (with read only permissions) in order to clone private repositories. -A SSH key is automatically generated and added to your main repository, but not to your submodules. -In order to give Read the Docs access to clone your submodules you'll need to add the public SSH key to each repository of your submodules. +Read the Docs uses SSH keys (with read only permissions) for GitLab and Bitbucket in order to clone private repositories, +this key is added to your main repository, but not to your submodules. +For GitHub we make use of a temporal token generated using our :ref:`GitHub App `. + +When a project is created, a SSH key is automatically generated. +You can use this SSH key to give Read the Docs access to clone your private submodules. .. note:: @@ -33,13 +36,6 @@ Since GitHub doesn't allow you to reuse a deploy key across different repositori you'll need to use `machine users `__ to give read access to several repositories using only one SSH key. -#. Remove the SSH deploy key that was added to the main repository on GitHub - - #. Go to your project on GitHub - #. Click on :guilabel:`Settings` - #. Click on :guilabel:`Deploy Keys` - #. Delete the key added by ``Read the Docs Commercial (readthedocs.com)`` - #. Create a GitHub user and give it read only permissions to all the necessary repositories. You can do this by adding the account as: diff --git a/docs/user/guides/pull-requests.rst b/docs/user/guides/pull-requests.rst index 97d57f55c32..f5df23997b5 100644 --- a/docs/user/guides/pull-requests.rst +++ b/docs/user/guides/pull-requests.rst @@ -4,15 +4,15 @@ How to configure pull request builds In this section, you can learn how to configure :doc:`pull request builds `. To enable pull request builds for your project, -your Read the Docs account needs to be connected to an account with a supported Git provider. +your Read the Docs project needs to be connected to a repository in a supported Git provider. See `Limitations`_ for more information. -If your account is already connected: +If your project is already connected: #. Go to your project dashboard -#. Go to :guilabel:`Admin`, then :guilabel:`Settings` +#. Go to :guilabel:`Settings`, then :guilabel:`Pull request builds` #. Enable the :guilabel:`Build pull requests for this project` option -#. Click on :guilabel:`Save` +#. Click on :guilabel:`Update` .. tip:: @@ -44,9 +44,9 @@ while private previews are only available to users with access to the Read the D To change the privacy level: #. Go to your project dashboard -#. Go to :guilabel:`Admin`, then :guilabel:`Settings` -#. Select your option in :guilabel:`Privacy level of builds from pull requests` -#. Click on :guilabel:`Save` +#. Go to :guilabel:`Settings`, then :guilabel:`Pull request builds` +#. Select your option in :guilabel:`Privacy level of builds of Pull Requests` +#. Click on :guilabel:`Update` Privacy levels work the same way as :ref:`normal versions `. @@ -54,8 +54,7 @@ Limitations ----------- - Pull requests are only available for **GitHub** and **GitLab** currently. Bitbucket is not yet supported. -- To enable this feature, your Read the Docs account needs to be connected to an - account with your Git provider. +- To enable this feature, your Read the Docs project needs to be connected to a repository in a supported Git provider. - Builds from pull requests have the same memory and time limitations :doc:`as regular builds `. - Additional formats like PDF aren't built in order to reduce build time. @@ -66,7 +65,10 @@ Troubleshooting --------------- No new builds are started when I open a pull request - The most common cause is that your repository's webhook is not configured to + The most common cause when using GitHub is that your Read the Docs project is not + connected to the corresponding repository on GitHub. + + The most common cause for GitLab and Bitbucket is that your repository's webhook is not configured to send Read the Docs pull request events. You'll need to re-sync your project's webhook integration to reconfigure the Read the Docs webhook. @@ -85,11 +87,6 @@ Build status is not being reported to your Git provider being updated with your Git provider, then your connected account may have out dated or insufficient permissions. - Make sure that you have granted access to the Read the Docs `GitHub OAuth App`_ for - your personal or organization GitHub account. - .. seealso:: - :ref:`guides/setup/git-repo-manual:Debugging webhooks` - :ref:`github-permission-troubleshooting` - -.. _GitHub OAuth App: https://github.com/settings/applications diff --git a/docs/user/guides/setup/git-repo-manual.rst b/docs/user/guides/setup/git-repo-manual.rst index b4df8b6b508..bfa454759ef 100644 --- a/docs/user/guides/setup/git-repo-manual.rst +++ b/docs/user/guides/setup/git-repo-manual.rst @@ -205,13 +205,7 @@ Webhook activation failed. Make sure you have the necessary permissions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you find this error, -make sure your user has permissions over the repository. -In case of GitHub, -check that you have granted access to the Read the Docs `OAuth App`_ to your organization. -A similar workflow is required for other supported providers. - -.. _OAuth App: https://github.com/settings/applications - +make sure your user has admin permissions over the repository. My project isn't automatically building ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/user/intro/add-project.rst b/docs/user/intro/add-project.rst index 8545a9053ec..7c5fa57b4b4 100644 --- a/docs/user/intro/add-project.rst +++ b/docs/user/intro/add-project.rst @@ -16,6 +16,7 @@ Automatically add your project #. Go to your :term:`dashboard`. #. Click on :guilabel:`Add project`. #. Type the name of the repository you want to add and click on it. + If you are using GitHub, make sure you have installed the :ref:`Read the Docs GitHub App ` in your repository. #. Click on :guilabel:`Continue`. #. Edit any of the pre-filled fields with information of the repository. #. Click on :guilabel:`Next`. diff --git a/docs/user/reference/git-integration.rst b/docs/user/reference/git-integration.rst index d2863677c80..bb025fd5fc5 100644 --- a/docs/user/reference/git-integration.rst +++ b/docs/user/reference/git-integration.rst @@ -12,8 +12,9 @@ Connecting your account provides the following features: See: :doc:`/intro/add-project`. โš™๏ธ Automatic configuration - Have your Git repository automatically configured with your Read the Docs :term:`webhook`, - which allows Read the Docs to build your docs on every change to your repository. + Have Read the Docs subscribe to your repository's events, + allowing us to build your docs on every change to your repository, + and keep in sync with your tags and branches. ๐Ÿšฅ๏ธ Commit status See your documentation build status as a commit status indicator on :doc:`pull request builds `. @@ -46,10 +47,17 @@ you can follow the :doc:`/intro/add-project` guide to actually add your project How automatic configuration works --------------------------------- -When your Read the Docs account is connected to |git_providers_or| and you :doc:`add a new Read the Docs project `: +When you Read the Docs account is connected to GitHub, and you :doc:`add a new Read the Docs project `: + +* Read the Docs automatically connects your project with the GitHub repository, + and subscribes to the repository's events. +* Read the Docs makes use of its :ref:`GitHub App ` to interact with your repository. + +When your Read the Docs account is connected to GitLab or Bitbucket, and you :doc:`add a new Read the Docs project `: * Read the Docs automatically creates a Read the Docs Integration that matches your Git provider. * Read the Docs creates an incoming webhook with your Git provider, which is automatically added to your Git repository's settings using the account connection. +* Read the Docs creates a deploy key for your Git repository, which is automatically added to your Git repository (when importing private repositories on |com_brand|). After project creation, you can continue to configure the project. @@ -64,7 +72,12 @@ including the ones that were automatically created. Read the Docs incoming webhook ------------------------------ -Accounts with |git_providers_and| integration automatically have Read the Docs' incoming :term:`webhook` configured on all Git repositories that are imported. +.. note:: + + When using GitHub, Read the Docs uses a GitHub App that subscribes to all required events. + You don't need to create a webhook on your repository. + +Accounts with GitLab and Bitbucket integrations automatically have Read the Docs' incoming :term:`webhook` configured on all repositories that are imported. Other setups can set up the webhook through :doc:`manual configuration `. When an incoming webhook notification is received, @@ -97,24 +110,30 @@ Read the Docs uses `OAuth`_ to connect to your account at |git_providers_or|. You are asked to grant permissions for Read the Docs to perform a number of actions on your behalf. At the same time, we use this process for authentication (login) -since we trust that |git_providers_or| have verified your user account and email address. +since we trust that the user who connects the account is the owner of Git provider account. By granting Read the Docs the requested permissions, we are issued a secret OAuth token from your Git provider. -Using the secret token, -we can automatically configure repositories during :doc:`project creation `. -We also use the token to send back build statuses and preview URLs for :doc:`pull requests `. +In the case of GitLab and Bitbucket, we can use the secret token +to automatically configure repositories during :doc:`project creation `, +for GitHub, you need to install our :ref:`GitHub App ` in the repository you want to import. .. _OAuth: https://en.wikipedia.org/wiki/OAuth .. note:: - Access granted to Read the Docs can always be revoked. - This is a function offered by all Git providers. + Access granted to Read the Docs can always be revoked. + This is a function offered by all Git providers. Git provider integrations ------------------------- +.. note:: + + When using GitHub, Read the Docs uses a GitHub App to interact with your repositories. + If the original user who connected the repository to Read the Docs loses access to the project or repository, + the GitHub App will still have access to the repository, and the integrations will continue to work. + If your project is using :doc:`Organizations ` (|com_brand|) or :term:`maintainers ` (|org_brand|), then you need to be aware of *who* is setting up the integration for the project. @@ -136,6 +155,40 @@ so that you can log in to Read the Docs with your connected account credentials. .. tab:: GitHub + Read the Docs requests the following permissions when connecting your Read the Docs account to GitHub. + + Account email addresses (read only) + We ask for this so we can verify your email address and create a Read the Docs account. + + When installing the Read the Docs GitHub App in a repository, you will be asked to grant the following permissions: + + Repository permissions + Commit statuses (read and write) + This allows Read the Docs to report the status of the build to GitHub. + Contents (read only) + This allows Read the Docs to clone the repository and build the documentation. + Metadata (read only) + This allows Read the Docs to read the repository collaborators and the permissions they have on the repository. + This is used to determine if the user can connect a repository to a Read the Docs project. + Pull requests (read and write) + This allows Read the Docs to subscribe to pull request events, + and to create a comment on the pull request with information about the build. + + Organization permissions + Members (read only) + This allows Read the Docs to read the organization members. + + + .. tab:: GitHub (old OAuth app integration) + + .. note:: + + Read the Docs used to use a GitHub OAuth application for integration, + which has been replaced by a `GitHub App `__. + If you haven't migrated your projects to the new GitHub App, + we will still use the OAuth application to interact with your repositories, + but we recommend migrating to the GitHub App for a better experience and more granular permissions. + Read the Docs requests the following permissions (more precisely, `OAuth scopes`_) when connecting your Read the Docs account to GitHub. @@ -197,35 +250,90 @@ so that you can log in to Read the Docs with your connected account credentials. * API access (``api``) which is needed to create webhooks in GitLab -.. _github-permission-troubleshooting: +GitHub App +---------- -GitHub permission troubleshooting ---------------------------------- +Read the Docs used to use a GitHub OAuth application for integration, +which has been replaced by a `GitHub App `__. +If you haven't migrated your projects to the new GitHub App, +we will still use the OAuth application similar to the other Git providers to interact with your repositories, +we recommend migrating to the GitHub App for a better experience and more granular permissions. -**Repositories not in your list to import**. +We have two GitHub Apps, one for each of our platforms: -Many organizations require approval for each OAuth application that is used, -or you might have disabled it in the past for your personal account. -This can happen at the personal or organization level, -depending on where the project you are trying to access has permissions from. +- `Read the Docs Community `__. +- `Read the Docs for Business `__. -.. tabs:: +Features +~~~~~~~~ + +When using GitHub, Read the Docs uses a GitHub App to interact with your repositories. +This has the following benefits over using an OAuth application (like the other Git providers): + +- More control over which repositories Read the Docs can access. + You don't need to grant access to all your repositories in order to create an account or import a single repository. +- No need to create webhooks on your repositories. + The GitHub App subscribes to all required events when you install it. +- No need to create a deploy key on your repository (|com_brand| only). + The GitHub App can clone your private repositories using a temporal token. +- If the original user who connected the repository to Read the Docs loses access to the project or repository, + the GitHub App will still have access to the repository. +- You can revoke access to the GitHub App at any time from your GitHub settings. +- Never out of sync with changes on your repository. + The GitHub App subscribes to all required events and will always keep your project up to date with your repository. + +Revoking access +~~~~~~~~~~~~~~~ + +You can revoke access to the Read the Docs GitHub App at any time from your GitHub settings. + +- `Read the Docs Community `__. +- `Read the Docs for Business `__. + +There are three ways to revoke access to the Read the Docs GitHub App: + +Revoke access to one or more repositories: + Remove the repositories from the list of repositories that the GitHub App has access to. +Suspend the GitHub App: + This will suspend the GitHub App and revoke access to all repositories. + The installation and configuration will still be available, + and you can re-enable the GitHub App at any time. +Uninstall the GitHub App: + This will uninstall the GitHub App and revoke access to all repositories. + The installation and configuration will be removed, + and you will need to re-install the GitHub App and reconfigure it to use it again. + +.. warning:: + + If you revoke access to the GitHub App with any of the above methods, + all projects linked to that repository will stop working, + but the projects and its documentation will still be available. + If you grant access to the repository again, + you will need to manually connect your project to the repository. + +.. _github-permission-troubleshooting: - .. tab:: Personal Account +Troubleshooting +~~~~~~~~~~~~~~~ - You need to make sure that you have granted access to the Read the Docs `OAuth App`_ to your **personal GitHub account**. - If you do not see Read the Docs in the `OAuth App`_ settings, you might need to disconnect and reconnect the GitHub service. +**Repository not in the list to import** - .. seealso:: GitHub docs on `requesting access to your personal OAuth`_ for step-by-step instructions. +Make sure you have installed the corresponding GitHub App in your GitHub account or organization, +and have granted access to the repository you want to import. - .. _OAuth App: https://github.com/settings/applications - .. _requesting access to your personal OAuth: https://docs.github.com/en/organizations/restricting-access-to-your-organizations-data/approving-oauth-apps-for-your-organization +- `Read the Docs Community `__. +- `Read the Docs for Business `__. - .. tab:: Organization Account +If you still can't see the repository in the list, +you may need to wait a couple of minutes and refresh the page, +or click on the "Refresh your repositories" button in the import page. - You need to make sure that you have granted access to the Read the Docs OAuth App to your **organization GitHub account**. - If you don't see "Read the Docs" listed, then you might need to connect GitHub to your social accounts as noted above. +**Repository is in the list, but can't be imported** - .. seealso:: GitHub doc on `requesting access to your organization OAuth`_ for step-by-step instructions. +Make sure you have admin access to the repository you are trying to import. +If you are using |org_brand|, make sure your project is public, +or use |com_brand| to import private repositories. - .. _requesting access to your organization OAuth: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/managing-your-membership-in-organizations/requesting-organization-approval-for-oauth-apps +If you still can't import the repository, +you may need to wait a couple of minutes and refresh the page, +or click on the "Refresh your repositories" button in the import page. diff --git a/docs/user/tutorial/index.rst b/docs/user/tutorial/index.rst index 8cd20098511..504d6b87678 100644 --- a/docs/user/tutorial/index.rst +++ b/docs/user/tutorial/index.rst @@ -70,14 +70,6 @@ On the authorization page, click the green :guilabel:`Authorize readthedocs` but GitHub authorization page -.. note:: - - Read the Docs needs elevated permissions to perform certain operations - that ensure that the workflow is as smooth as possible, - like installing :term:`webhooks `. - If you want to learn more, - check out :ref:`reference/git-integration:permissions for connected accounts`. - After that, you will be redirected to Read the Docs to confirm your e-mail and username. Click the :guilabel:`Sign Up ยป` button to create your account and open your :term:`dashboard`. @@ -95,16 +87,11 @@ Importing the project to Read the Docs To import your GitHub project to Read the Docs: -#. Click the :guilabel:`Import a Project` button on your `dashboard `_. +#. Click the :guilabel:`Add project` button on your `dashboard `_. -#. Click the |:heavy_plus_sign:| button to the right of your ``rtd-tutorial`` project. If the list of repositories is empty, click the |:arrows_counterclockwise:| button. +#. Click on :guilabel:`Install GitHub App on repository`, and choose your account and select the repository you created in the previous step. - .. figure:: /_static/images/tutorial/rtd-import-projects.gif - :width: 80% - :align: center - :alt: Import projects workflow - - Import projects workflow +#. Type the repository name in the search box, and select the repository from the list, and click on :guilabel:`Continue`. #. Enter some details about your Read the Docs project: @@ -113,9 +100,6 @@ To import your GitHub project to Read the Docs: so it is better if you prepend your username, for example ``{username}-rtd-tutorial``. - Repository URL - The URL that contains the documentation source. Leave the automatically filled value. - Default branch Name of the default branch of the project, leave it as ``main``. @@ -175,7 +159,7 @@ Configuring the project To update the project description and configure the notification settings: -#. Navigate back to the :term:`project page` and click the :guilabel:`โš™ Admin` button,to open the Settings page. +#. Navigate back to the :term:`project page` and click the :guilabel:`โš™ Settings` button, to open the settings page. #. Update the project description by adding the following text: @@ -194,7 +178,7 @@ and show you a preview of the documentation with those changes. To trigger builds from pull requests: -#. Click the :guilabel:`Settings` link on the left under the :guilabel:`โš™ Admin` menu, check the "Build pull requests for this project" checkbox, and click the :guilabel:`Save` button at the bottom of the page. +#. Click the :guilabel:`Pull request builds` link on the left under the :guilabel:`โš™ Settings` menu, check the "Build pull requests for this project" checkbox, and click the :guilabel:`Update` button at the bottom of the page. #. Make some changes to your documentation: diff --git a/readthedocs/api/v2/serializers.py b/readthedocs/api/v2/serializers.py index 0348cc159a2..9a2fe27edb8 100644 --- a/readthedocs/api/v2/serializers.py +++ b/readthedocs/api/v2/serializers.py @@ -93,6 +93,7 @@ class Meta(ProjectSerializer.Meta): "environment_variables", "max_concurrent_builds", "readthedocs_yaml_path", + "clone_token", ) diff --git a/readthedocs/oauth/migrate.py b/readthedocs/oauth/migrate.py new file mode 100644 index 00000000000..addb0dd262c --- /dev/null +++ b/readthedocs/oauth/migrate.py @@ -0,0 +1,329 @@ +"""This module contains the logic to help users migrate from the GitHub OAuth App to the GitHub App.""" + +from dataclasses import dataclass + +from allauth.socialaccount.models import SocialAccount +from allauth.socialaccount.providers.github.provider import GitHubProvider +from django.conf import settings + +from readthedocs.allauth.providers.githubapp.provider import GitHubAppProvider +from readthedocs.core.permissions import AdminPermission +from readthedocs.integrations.models import Integration +from readthedocs.oauth.constants import GITHUB +from readthedocs.oauth.constants import GITHUB_APP +from readthedocs.oauth.models import GitHubAccountType +from readthedocs.oauth.models import RemoteRepository +from readthedocs.oauth.services import GitHubAppService +from readthedocs.oauth.services import GitHubService +from readthedocs.projects.models import Project + + +@dataclass +class GitHubAccountTarget: + login: str + id: int + type: GitHubAccountType + + +@dataclass +class InstallationTargetGroup: + """Group of repositories that should be installed in the same target (user or organization).""" + + target_id: int + target_type: GitHubAccountType + target_name: str + repository_ids: set[int] + + @property + def link(self): + """ + Create a link to install the GitHub App on the target with the required repositories pre-selected. + + See https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/migrating-oauth-apps-to-github-apps#prompt-users-to-install-your-github-app. + """ + repository_ids = [] + for repository_id in self.repository_ids: + repository_ids.append(f"&repository_ids[]={repository_id}") + repository_ids = "".join(repository_ids) + + base_url = ( + f"https://github.com/apps/{settings.GITHUB_APP_NAME}/installations/new/permissions" + ) + return f"{base_url}?suggested_target_id={self.target_id}{repository_ids}" + + @property + def installed(self): + """ + Check if the app was already installed on the target. + + If we don't have any repositories left to install, the app was already installed, + or we don't have any repositories to install the app on. + """ + return not bool(self.repository_ids) + + +@dataclass +class MigrationTarget: + """Information about an individual project that needs to be migrated.""" + + project: Project + has_installation: bool + is_admin: bool + target_id: int + + @property + def installation_link(self): + """ + Create a link to install the GitHub App on the target repository. + + See https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/migrating-oauth-apps-to-github-apps + """ + base_url = ( + f"https://github.com/apps/{settings.GITHUB_APP_NAME}/installations/new/permissions" + ) + return f"{base_url}?suggested_target_id={self.target_id}&repository_ids[]={self.project.remote_repository.remote_id}" + + @property + def can_be_migrated(self): + """ + Check if the project can be migrated. + + The project can be migrated if the user is an admin on the repository and the GitHub App is installed. + """ + return self.is_admin and self.has_installation + + +@dataclass +class MigrationResult: + """Result of a migration operation.""" + + webhook_removed: bool + ssh_key_removed: bool + + +class MigrationError(Exception): + """Error raised when a migration operation fails.""" + + pass + + +def get_installation_target_groups_for_user(user) -> list[InstallationTargetGroup]: + """Get all targets (accounts and organizations) that the user needs to install the GitHub App on.""" + # Since we don't save the ID of the owner of each repository, we group all repositories + # that we aren't able to identify the owner into the user's account. + # GitHub will ignore the repositories that the user doesn't own. + default_target_account = _get_default_github_account_target(user) + + targets = {} + for project, has_intallation, _ in _get_projects_missing_migration(user): + remote_repository = project.remote_repository + target_account = _get_github_account_target(remote_repository) or default_target_account + if target_account.id not in targets: + targets[target_account.id] = InstallationTargetGroup( + target_id=target_account.id, + target_name=target_account.login, + target_type=target_account.type, + repository_ids=set(), + ) + if not has_intallation: + targets[target_account.id].repository_ids.add(int(remote_repository.remote_id)) + + # Include accounts that have already migrated projects, + # so they are shown as "Installed" in the UI. + for project in get_migrated_projects(user): + remote_repository = project.remote_repository + target_account = _get_github_account_target(remote_repository) or default_target_account + if target_account.id not in targets: + targets[target_account.id] = InstallationTargetGroup( + target_id=target_account.id, + target_name=target_account.login, + target_type=GitHubAccountType.USER, + repository_ids=set(), + ) + + return list(targets.values()) + + +def _get_default_github_account_target(user): + # NOTE: there are some users that have more than one GH account connected. + # They will need to migrate each account at a time. + account = user.socialaccount_set.filter(provider=GitHubProvider.id).first() + if not account: + account = user.socialaccount_set.filter(provider=GitHubAppProvider.id).first() + + return GitHubAccountTarget( + login=account.extra_data.get("login", "ghost"), + id=int(account.uid), + type=GitHubAccountType.USER, + ) + + +def _get_github_account_target(remote_repository): + """ + Get the GitHub account target for a repository. + + This will return the account that owns the repository, if we can identify it. + For repositories owned by organizations, we return the organization account, + for repositories owned by users, we try to guess the account based on the repository owner + (as we don't save the owner ID in the repository). + """ + if remote_repository.organization: + return GitHubAccountTarget( + login=remote_repository.organization.slug, + id=int(remote_repository.organization.remote_id), + type=GitHubAccountType.ORGANIZATION, + ) + login = remote_repository.full_name.split("/", 1)[0] + account = SocialAccount.objects.filter( + provider__in=[GitHubProvider.id, GitHubAppProvider.id], extra_data__login=login + ).first() + if account: + return GitHubAccountTarget( + login=login, + id=int(account.uid), + type=GitHubAccountType.USER, + ) + return None + + +def _get_projects_missing_migration(user): + """ + Get all projects where the user has admin permissions that are still connected to the old GitHub OAuth App. + + Returns a generator with the project, a boolean indicating if the GitHub App is installed on the repository, + and a boolean indicating if the user has admin permissions on the repository. + """ + projects = ( + AdminPermission.projects(user, admin=True) + .filter(remote_repository__vcs_provider=GITHUB) + .select_related( + "remote_repository", + "remote_repository__organization", + ) + ) + for project in projects: + remote_repository = project.remote_repository + has_installation = RemoteRepository.objects.filter( + remote_id=remote_repository.remote_id, + vcs_provider=GITHUB_APP, + github_app_installation__isnull=False, + ).exists() + is_admin = ( + RemoteRepository.objects.for_project_linking(user) + .filter( + remote_id=project.remote_repository.remote_id, + vcs_provider=GITHUB_APP, + github_app_installation__isnull=False, + ) + .exists() + ) + yield project, has_installation, is_admin + + +def get_migrated_projects(user): + return ( + AdminPermission.projects(user, admin=True) + .filter(remote_repository__vcs_provider=GITHUB_APP) + .select_related( + "remote_repository", + ) + ) + + +def get_valid_projects_missing_migration(user): + for project, has_installation, is_admin in _get_projects_missing_migration(user): + if has_installation and is_admin: + yield project + + +def get_migration_targets(user) -> list[MigrationTarget]: + """Get all projects that the user needs to migrate to the GitHub App.""" + targets = [] + default_target_account = _get_default_github_account_target(user) + for project, has_installation, is_admin in _get_projects_missing_migration(user): + remote_repository = project.remote_repository + target_account = _get_github_account_target(remote_repository) or default_target_account + targets.append( + MigrationTarget( + project=project, + has_installation=has_installation, + is_admin=is_admin, + target_id=target_account.id, + ) + ) + return targets + + +def get_old_app_link(): + """ + Get the link to the old GitHub OAuth App settings page. + + Useful so users can revoke the old app. + """ + client_id = settings.SOCIALACCOUNT_PROVIDERS["github"]["APPS"][0]["client_id"] + return f"https://github.com/settings/connections/applications/{client_id}" + + +def migrate_project_to_github_app(project, user) -> MigrationResult: + """ + Migrate a project to the new GitHub App. + + This will remove the webhook and SSH key from the old GitHub OAuth App and + connect the project to the new GitHub App. + + Returns a MigrationResult with the status of the migration. + Raises a MigrationError if the project can't be migrated, + this should never happen as we don't allow migrating projects + that can't be migrated from the UI. + """ + # No remote repository, nothing to migrate. + if not project.remote_repository: + raise MigrationError("Project isn't connected to a repository") + + service_class = project.get_git_service_class() + + # Already migrated, nothing to do. + if service_class == GitHubAppService: + return MigrationResult(webhook_removed=True, ssh_key_removed=True) + + # Not a GitHub project, nothing to migrate. + if service_class != GitHubService: + raise MigrationError("Project isn't connected to a GitHub repository") + + new_remote_repository = RemoteRepository.objects.filter( + remote_id=project.remote_repository.remote_id, + vcs_provider=GITHUB_APP, + github_app_installation__isnull=False, + ).first() + + if not new_remote_repository: + raise MigrationError("You need to install the GitHub App on the repository") + + new_remote_repository = ( + RemoteRepository.objects.for_project_linking(user) + .filter( + remote_id=project.remote_repository.remote_id, + vcs_provider=GITHUB_APP, + github_app_installation__isnull=False, + ) + .first() + ) + if not new_remote_repository: + raise MigrationError("You must have admin permissions on the repository to migrate it") + + webhook_removed = False + ssh_key_removed = False + for service in service_class.for_project(project): + if not webhook_removed and service.remove_webhook(project): + webhook_removed = True + + if not ssh_key_removed and service.remove_ssh_key(project): + ssh_key_removed = True + + project.integrations.filter(integration_type=Integration.GITHUB_WEBHOOK).delete() + project.remote_repository = new_remote_repository + project.save() + return MigrationResult( + webhook_removed=webhook_removed, + ssh_key_removed=ssh_key_removed, + ) diff --git a/readthedocs/oauth/notifications.py b/readthedocs/oauth/notifications.py index 968c1be8ebd..496a7c83c9b 100644 --- a/readthedocs/oauth/notifications.py +++ b/readthedocs/oauth/notifications.py @@ -5,6 +5,7 @@ from django.utils.translation import gettext_lazy as _ from readthedocs.notifications.constants import ERROR +from readthedocs.notifications.constants import WARNING from readthedocs.notifications.messages import Message from readthedocs.notifications.messages import registry @@ -14,6 +15,8 @@ MESSAGE_OAUTH_WEBHOOK_INVALID = "oauth:webhook:invalid" MESSAGE_OAUTH_BUILD_STATUS_FAILURE = "oauth:status:send-failed" MESSAGE_OAUTH_DEPLOY_KEY_ATTACHED_FAILED = "oauth:deploy-key:attached-failed" +MESSAGE_OAUTH_WEBHOOK_NOT_REMOVED = "oauth:migration:webhook-not-removed" +MESSAGE_OAUTH_DEPLOY_KEY_NOT_REMOVED = "oauth:migration:ssh-key-not-removed" messages = [ Message( @@ -83,5 +86,31 @@ ), type=ERROR, ), + Message( + id=MESSAGE_OAUTH_WEBHOOK_NOT_REMOVED, + header=_("Failed to remove webhook"), + body=_( + textwrap.dedent( + """ + Failed to remove webhook from the {{ repo_full_name }} repository, please remove it manually + from the repository settings (search for a webhook containing "{{ project_slug }}" in the URL). + """ + ).strip(), + ), + type=WARNING, + ), + Message( + id=MESSAGE_OAUTH_DEPLOY_KEY_NOT_REMOVED, + header=_("Failed to remove deploy key"), + body=_( + textwrap.dedent( + """ + Failed to remove deploy key from the {{ repo_full_name }} repository, please remove it manually + from the repository settings (search for a deploy key containing "{{ project_slug }}" in the title). + """ + ) + ), + type=WARNING, + ), ] registry.add(messages) diff --git a/readthedocs/oauth/querysets.py b/readthedocs/oauth/querysets.py index 9dffc478078..573d7f12ae1 100644 --- a/readthedocs/oauth/querysets.py +++ b/readthedocs/oauth/querysets.py @@ -3,6 +3,7 @@ from django.db import models from readthedocs.core.querysets import NoReprQuerySet +from readthedocs.oauth.constants import GITHUB from readthedocs.oauth.constants import GITHUB_APP @@ -14,9 +15,12 @@ def api(self, user=None): if not user.is_authenticated: return self.none() queryset = self.filter(users=user) - # TODO: Once we are migrated into GitHub App we should include these repositories/organizations. - # Exclude repositories/organizations from the GitHub App for now to avoid duplicated entries. - queryset = queryset.exclude(vcs_provider=GITHUB_APP) + + # If the user has already started using the GitHub App, + # we shouldn't show repositories from the old GitHub integration. + if queryset.filter(vcs_provider=GITHUB_APP).exists(): + queryset = queryset.exclude(vcs_provider=GITHUB) + return queryset def api_v2(self, *args, **kwargs): @@ -37,9 +41,12 @@ def for_project_linking(self, user): remote_repository_relations__user=user, remote_repository_relations__admin=True, ) - # TODO: Once we are migrated into GitHub App we should include these repositories/organizations. - # Exclude repositories/organizations from the GitHub App for now to avoid duplicated entries. - queryset = queryset.exclude(vcs_provider=GITHUB_APP) + + # If the user has already started using the GitHub App, + # we shouldn't show repositories from the old GitHub integration. + if queryset.filter(vcs_provider=GITHUB_APP).exists(): + queryset = queryset.exclude(vcs_provider=GITHUB) + return queryset.distinct() diff --git a/readthedocs/oauth/services/github.py b/readthedocs/oauth/services/github.py index 8863afacc13..829029f8e66 100644 --- a/readthedocs/oauth/services/github.py +++ b/readthedocs/oauth/services/github.py @@ -406,6 +406,42 @@ def update_webhook(self, project, integration): return (False, resp) + def remove_webhook(self, project): + owner, repo = build_utils.get_github_username_repo(url=project.repo) + + try: + resp = self.session.get(f"{self.base_api_url}/repos/{owner}/{repo}/hooks") + resp.raise_for_status() + data = resp.json() + except Exception: + log.info("Failed to get GitHub webhooks for project.") + return False + + hook_targets = [ + f"{settings.PUBLIC_API_URL}/api/v2/webhook/{project.slug}/", + f"{settings.PUBLIC_API_URL}/api/v2/webhook/github/{project.slug}/", + ] + if "app." in settings.PUBLIC_API_URL: + hook_targets.append(hook_targets[0].replace("app.", "", 1)) + hook_targets.append(hook_targets[1].replace("app.", "", 1)) + + for hook in data: + hook_url = hook["config"]["url"] + for hook_target in hook_targets: + if hook_url.startswith(hook_target): + try: + self.session.delete( + f"{self.base_api_url}/repos/{owner}/{repo}/hooks/{hook['id']}" + ).raise_for_status() + except Exception: + log.info("Failed to remove GitHub webhook for project.") + return False + return True + + def remove_ssh_key(self, project): + # Overridden in corporate + return True + def send_build_status(self, *, build, commit, status): """ Create GitHub commit status for project. diff --git a/readthedocs/profiles/tests/__init__.py b/readthedocs/profiles/tests/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/readthedocs/profiles/tests/test_views.py b/readthedocs/profiles/tests/test_views.py new file mode 100644 index 00000000000..be62de0867a --- /dev/null +++ b/readthedocs/profiles/tests/test_views.py @@ -0,0 +1,684 @@ +from unittest import mock + +import pytest +import requests_mock +from allauth.socialaccount.models import SocialAccount, SocialToken +from allauth.socialaccount.providers.github.provider import GitHubProvider +from django.conf import settings +from django.contrib.auth.models import User +from django.test import TestCase, override_settings +from django.urls import reverse +from django_dynamic_fixture import get + +from readthedocs.allauth.providers.githubapp.provider import GitHubAppProvider +from readthedocs.notifications.models import Notification +from readthedocs.oauth.constants import GITHUB, GITHUB_APP +from readthedocs.oauth.migrate import InstallationTargetGroup, MigrationTarget +from readthedocs.oauth.models import ( + GitHubAccountType, + GitHubAppInstallation, + RemoteOrganization, + RemoteOrganizationRelation, + RemoteRepository, + RemoteRepositoryRelation, +) +from readthedocs.oauth.notifications import ( + MESSAGE_OAUTH_DEPLOY_KEY_NOT_REMOVED, + MESSAGE_OAUTH_WEBHOOK_NOT_REMOVED, +) +from readthedocs.oauth.services.github import GitHubService +from readthedocs.projects.models import Project + + +@pytest.mark.skipif( + not settings.RTD_EXT_THEME_ENABLED, reason="Not applicable for the old theme" +) +@override_settings(GITHUB_APP_NAME="readthedocs") +class TestMigrateToGitHubAppView(TestCase): + def setUp(self): + self.user = get(User) + self.social_account_github = get( + SocialAccount, + provider=GitHubProvider.id, + user=self.user, + uid="1234", + extra_data={"login": "user"}, + ) + get( + SocialToken, + account=self.social_account_github, + ) + self.social_account_github_app = get( + SocialAccount, + provider=GitHubAppProvider.id, + user=self.user, + uid="1234", + extra_data={"login": "user"}, + ) + self.github_app_installation = get( + GitHubAppInstallation, + installation_id=1111, + target_id=int(self.social_account_github_app.uid), + target_type=GitHubAccountType.USER, + ) + + # Project with remote repository where the user is admin. + self.remote_repository_a = get( + RemoteRepository, + name="repo-a", + full_name="user/repo-a", + html_url="https://github.com/user/repo-a", + remote_id="1111", + vcs_provider=GITHUB, + ) + get( + RemoteRepositoryRelation, + user=self.user, + account=self.social_account_github, + remote_repository=self.remote_repository_a, + admin=True, + ) + self.project_with_remote_repository = get( + Project, + users=[self.user], + remote_repository=self.remote_repository_a, + ) + + # Project with remote repository where the user is not admin. + self.remote_repository_b = get( + RemoteRepository, + name="repo-b", + full_name="user/repo-b", + html_url="https://github.com/user/repo-b", + remote_id="2222", + vcs_provider=GITHUB, + ) + get( + RemoteRepositoryRelation, + user=self.user, + account=self.social_account_github, + remote_repository=self.remote_repository_b, + admin=False, + ) + self.project_with_remote_repository_no_admin = get( + Project, + users=[self.user], + remote_repository=self.remote_repository_b, + ) + + # Project with remote repository where the user doesn't have permissions at all. + self.remote_repository_c = get( + RemoteRepository, + name="repo-c", + full_name="user2/repo-c", + html_url="https://github.com/user2/repo-c", + remote_id="3333", + vcs_provider=GITHUB, + ) + get( + RemoteRepositoryRelation, + user=self.user, + account=self.social_account_github, + remote_repository=self.remote_repository_c, + admin=False, + ) + self.project_with_remote_repository_no_member = get( + Project, + users=[self.user], + remote_repository=self.remote_repository_c, + ) + + # Project connected to a remote repository that belongs to an organization. + self.remote_organization = get( + RemoteOrganization, + slug="org", + name="Organization", + remote_id="9999", + vcs_provider=GITHUB, + ) + get( + RemoteOrganizationRelation, + user=self.user, + account=self.social_account_github, + ) + self.remote_repository_d = get( + RemoteRepository, + name="repo-d", + full_name="org/repo-d", + html_url="https://github.com/org/repo-d", + remote_id="4444", + organization=self.remote_organization, + ) + get( + RemoteRepositoryRelation, + user=self.user, + account=self.social_account_github, + remote_repository=self.remote_repository_d, + admin=True, + ) + self.project_with_remote_organization = get( + Project, + users=[self.user], + remote_repository=self.remote_repository_d, + ) + self.github_app_organization_installation = get( + GitHubAppInstallation, + installation_id=2222, + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + ) + + # Project without a remote repository. + self.project_without_remote_repository = get( + Project, + users=[self.user], + repo="https://github.com/user/repo-e", + ) + + self.url = reverse("migrate_to_github_app") + self.client.force_login(self.user) + + def _create_github_app_remote_repository(self, remote_repository): + new_remote_repository = get( + RemoteRepository, + name=remote_repository.name, + full_name=remote_repository.full_name, + html_url=remote_repository.html_url, + remote_id=remote_repository.remote_id, + vcs_provider=GITHUB_APP, + github_app_installation=self.github_app_installation, + ) + if remote_repository.organization: + new_remote_repository.organization = get( + RemoteOrganization, + slug=remote_repository.organization.slug, + name=remote_repository.organization.name, + remote_id=remote_repository.organization.remote_id, + vcs_provider=GITHUB_APP, + ) + new_remote_repository.github_app_installation = ( + self.github_app_organization_installation + ) + new_remote_repository.save() + for relation in remote_repository.remote_repository_relations.all(): + github_app_account = relation.user.socialaccount_set.get( + provider=GitHubAppProvider.id + ) + get( + RemoteRepositoryRelation, + user=relation.user, + account=github_app_account, + remote_repository=new_remote_repository, + admin=relation.admin, + ) + return new_remote_repository + + def test_user_without_github_account(self): + self.user.socialaccount_set.all().delete() + response = self.client.get(self.url) + assert response.status_code == 302 + response = self.client.get(reverse("projects_dashboard")) + content = response.content.decode() + print(content) + assert ( + "You don\\u0026#x27\\u003Bt have any GitHub account connected." in content + ) + + def test_user_without_github_account_but_with_github_app_account(self): + self.user.socialaccount_set.exclude(provider=GitHubAppProvider.id).delete() + response = self.client.get(self.url) + assert response.status_code == 302 + response = self.client.get(reverse("projects_dashboard")) + assert ( + "You have already migrated your account to the new GitHub App" + in response.content.decode() + ) + + @requests_mock.Mocker(kw="request") + def test_migration_page_initial_state(self, request): + request.get("https://api.github.com/user", status_code=200) + + self.user.socialaccount_set.filter(provider=GitHubAppProvider.id).delete() + response = self.client.get(self.url) + assert response.status_code == 200 + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is False + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={1111, 2222, 3333}, + ), + InstallationTargetGroup( + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + target_name="org", + repository_ids={4444}, + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_organization, + has_installation=False, + is_admin=False, + target_id=int(self.remote_organization.remote_id), + ), + ] + assert list(context["migrated_projects"]) == [] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is False + assert context["old_github_account"] == self.social_account_github + + @requests_mock.Mocker(kw="request") + def test_migration_page_step_connect_done(self, request): + request.get("https://api.github.com/user", status_code=200) + response = self.client.get(self.url) + assert response.status_code == 200 + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is True + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={1111, 2222, 3333}, + ), + InstallationTargetGroup( + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + target_name="org", + repository_ids={4444}, + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_organization, + has_installation=False, + is_admin=False, + target_id=int(self.remote_organization.remote_id), + ), + ] + assert list(context["migrated_projects"]) == [] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is False + assert context["old_github_account"] == self.social_account_github + + @requests_mock.Mocker(kw="request") + def test_migration_page_step_install_done(self, request): + request.get("https://api.github.com/user", status_code=200) + + self._create_github_app_remote_repository(self.remote_repository_a) + self._create_github_app_remote_repository(self.remote_repository_b) + self._create_github_app_remote_repository(self.remote_repository_d) + + response = self.client.get(self.url) + assert response.status_code == 200 + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is True + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={3333}, + ), + InstallationTargetGroup( + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + target_name="org", + repository_ids=set(), + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository, + has_installation=True, + is_admin=True, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=True, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_organization, + has_installation=True, + is_admin=True, + target_id=int(self.remote_organization.remote_id), + ), + ] + assert list(context["migrated_projects"]) == [] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is False + assert context["old_github_account"] == self.social_account_github + + @requests_mock.Mocker(kw="request") + @mock.patch.object(GitHubService, "remove_webhook") + @mock.patch.object(GitHubService, "remove_ssh_key") + def test_migration_page_step_migrate_one_project( + self, remove_ssh_key, remove_webhook, request + ): + request.get("https://api.github.com/user", status_code=200) + + remove_ssh_key.return_value = True + remove_webhook.return_value = True + + self._create_github_app_remote_repository(self.remote_repository_a) + self._create_github_app_remote_repository(self.remote_repository_b) + self._create_github_app_remote_repository(self.remote_repository_d) + + response = self.client.post( + self.url, data={"project": self.project_with_remote_repository.slug} + ) + assert response.status_code == 302 + response = self.client.get(self.url) + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is True + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={3333}, + ), + InstallationTargetGroup( + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + target_name="org", + repository_ids=set(), + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=True, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_organization, + has_installation=True, + is_admin=True, + target_id=int(self.remote_organization.remote_id), + ), + ] + assert list(context["migrated_projects"]) == [ + self.project_with_remote_repository, + ] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is False + assert context["old_github_account"] == self.social_account_github + + @requests_mock.Mocker(kw="request") + @mock.patch.object(GitHubService, "remove_webhook") + @mock.patch.object(GitHubService, "remove_ssh_key") + def test_migration_page_step_migrate_all_projects( + self, remove_ssh_key, remove_webhook, request + ): + request.get("https://api.github.com/user", status_code=200) + + remove_ssh_key.return_value = True + remove_webhook.return_value = True + + self._create_github_app_remote_repository(self.remote_repository_a) + self._create_github_app_remote_repository(self.remote_repository_b) + self._create_github_app_remote_repository(self.remote_repository_d) + + response = self.client.post(self.url) + assert response.status_code == 302 + response = self.client.get(self.url) + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is True + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={3333}, + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=True, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + ] + assert list(context["migrated_projects"]) == [ + self.project_with_remote_repository, + self.project_with_remote_organization, + ] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is False + assert context["old_github_account"] == self.social_account_github + + @requests_mock.Mocker(kw="request") + @mock.patch.object(GitHubService, "remove_webhook") + @mock.patch.object(GitHubService, "remove_ssh_key") + def test_migration_page_step_migrate_one_project_with_errors( + self, remove_ssh_key, remove_webhook, request + ): + request.get("https://api.github.com/user", status_code=200) + + remove_ssh_key.return_value = False + remove_webhook.return_value = False + + self._create_github_app_remote_repository(self.remote_repository_a) + self._create_github_app_remote_repository(self.remote_repository_b) + self._create_github_app_remote_repository(self.remote_repository_d) + + response = self.client.post( + self.url, data={"project": self.project_with_remote_repository.slug} + ) + assert response.status_code == 302 + response = self.client.get(self.url) + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is True + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={3333}, + ), + InstallationTargetGroup( + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + target_name="org", + repository_ids=set(), + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=True, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_organization, + has_installation=True, + is_admin=True, + target_id=int(self.remote_organization.remote_id), + ), + ] + assert list(context["migrated_projects"]) == [ + self.project_with_remote_repository, + ] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is False + assert context["old_github_account"] == self.social_account_github + + notifications = Notification.objects.for_user(self.user, self.user) + assert notifications.count() == 2 + assert notifications.filter( + message_id=MESSAGE_OAUTH_WEBHOOK_NOT_REMOVED + ).exists() + assert notifications.filter( + message_id=MESSAGE_OAUTH_DEPLOY_KEY_NOT_REMOVED + ).exists() + + @requests_mock.Mocker(kw="request") + def test_migration_page_step_revoke_done(self, request): + request.get("https://api.github.com/user", status_code=401) + response = self.client.get(self.url) + assert response.status_code == 200 + context = response.context + + assert context["step"] == "overview" + assert context["has_multiple_github_accounts"] is False + assert context["step_connect_completed"] is True + assert context["installation_target_groups"] == [ + InstallationTargetGroup( + target_id=int(self.social_account_github.uid), + target_type=GitHubAccountType.USER, + target_name="user", + repository_ids={1111, 2222, 3333}, + ), + InstallationTargetGroup( + target_id=int(self.remote_organization.remote_id), + target_type=GitHubAccountType.ORGANIZATION, + target_name="org", + repository_ids={4444}, + ), + ] + assert context["github_app_name"] == "readthedocs" + assert context["migration_targets"] == [ + MigrationTarget( + project=self.project_with_remote_repository, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_admin, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_repository_no_member, + has_installation=False, + is_admin=False, + target_id=int(self.social_account_github.uid), + ), + MigrationTarget( + project=self.project_with_remote_organization, + has_installation=False, + is_admin=False, + target_id=int(self.remote_organization.remote_id), + ), + ] + assert list(context["migrated_projects"]) == [] + assert ( + context["old_application_link"] + == "https://github.com/settings/connections/applications/123" + ) + assert context["step_revoke_completed"] is True + assert context["old_github_account"] == self.social_account_github diff --git a/readthedocs/profiles/urls/private.py b/readthedocs/profiles/urls/private.py index 597019549fc..f65651a1421 100644 --- a/readthedocs/profiles/urls/private.py +++ b/readthedocs/profiles/urls/private.py @@ -41,6 +41,11 @@ views.AccountAdvertisingEdit.as_view(), name="account_advertising", ), + path( + "migrate-to-github-app/", + views.MigrateToGitHubAppView.as_view(), + name="migrate_to_github_app", + ), ] urlpatterns += account_urls diff --git a/readthedocs/profiles/views.py b/readthedocs/profiles/views.py index ef51566a028..93ef7edbc57 100644 --- a/readthedocs/profiles/views.py +++ b/readthedocs/profiles/views.py @@ -1,7 +1,11 @@ """Views for creating, editing and viewing site-specific user profiles.""" +from enum import StrEnum +from enum import auto + from allauth.account.views import LoginView as AllAuthLoginView from allauth.account.views import LogoutView as AllAuthLogoutView +from allauth.socialaccount.providers.github.provider import GitHubProvider from django.conf import settings from django.contrib import messages from django.contrib.auth import logout @@ -18,8 +22,10 @@ from vanilla import DetailView from vanilla import FormView from vanilla import ListView +from vanilla import TemplateView from vanilla import UpdateView +from readthedocs.allauth.providers.githubapp.provider import GitHubAppProvider from readthedocs.audit.filters import UserSecurityLogFilter from readthedocs.audit.models import AuditLog from readthedocs.core.forms import UserAdvertisingForm @@ -30,6 +36,16 @@ from readthedocs.core.models import UserProfile from readthedocs.core.permissions import AdminPermission from readthedocs.core.utils.extend import SettingsOverrideObject +from readthedocs.notifications.models import Notification +from readthedocs.oauth.clients import get_oauth2_client +from readthedocs.oauth.migrate import get_installation_target_groups_for_user +from readthedocs.oauth.migrate import get_migrated_projects +from readthedocs.oauth.migrate import get_migration_targets +from readthedocs.oauth.migrate import get_old_app_link +from readthedocs.oauth.migrate import get_valid_projects_missing_migration +from readthedocs.oauth.migrate import migrate_project_to_github_app +from readthedocs.oauth.notifications import MESSAGE_OAUTH_DEPLOY_KEY_NOT_REMOVED +from readthedocs.oauth.notifications import MESSAGE_OAUTH_WEBHOOK_NOT_REMOVED from readthedocs.organizations.models import Organization from readthedocs.projects.models import Project from readthedocs.projects.utils import get_csv_file @@ -278,3 +294,133 @@ def get_queryset(self): queryset=queryset, ) return self.filter.qs + + +class MigrationSteps(StrEnum): + overview = auto() + connect = auto() + install = auto() + migrate = auto() + revoke = auto() + disconnect = auto() + + +class MigrateToGitHubAppView(PrivateViewMixin, TemplateView): + """ + View to help users migrate their account to the new GitHub App. + + This view will guide the user through the process of migrating their account + and projects to the new GitHub App. + + A get request will show the overview of the migration process, + and each step to follow. A post request will migrate a single project + if the project slug is provided in the request, otherwise, it will migrate + all projects that can be migrated. + + In case we weren't able to remove the webhook or SSH key from the old GitHub App, + we create a notification for the user, so they can manually remove it. + """ + + template_name = "profiles/private/migrate_to_gh_app.html" + + def get(self, request, *args, **kwargs): + if self._get_old_github_account() is None: + if self._get_new_github_account(): + msg = _("You have already migrated your account to the new GitHub App.") + else: + msg = _("You don't have any GitHub account connected.") + messages.info(request, msg) + return HttpResponseRedirect(reverse("homepage")) + + return super().get(request, *args, **kwargs) + + def get_context_data(self, **kwargs): + context = super().get_context_data(**kwargs) + + step = self.request.GET.get("step", MigrationSteps.overview) + if step not in MigrationSteps: + step = MigrationSteps.overview + context["step"] = step + + user = self.request.user + + context["has_multiple_github_accounts"] = ( + user.socialaccount_set.filter(provider=GitHubProvider.id).count() > 1 + ) + context["step_connect_completed"] = self._has_new_account_for_old_account() + context["installation_target_groups"] = get_installation_target_groups_for_user(user) + context["github_app_name"] = settings.GITHUB_APP_NAME + context["migration_targets"] = get_migration_targets(user) + context["migrated_projects"] = get_migrated_projects(user) + context["old_application_link"] = get_old_app_link() + context["step_revoke_completed"] = self._is_access_to_old_github_account_revoked() + context["old_github_account"] = self._get_old_github_account() + # NOTE: this is a done, so the template can display this single element in a list. + context["old_github_accounts"] = [context["old_github_account"]] + return context + + def _is_access_to_old_github_account_revoked(self): + old_account = self._get_old_github_account() + if not old_account: + return True + client = get_oauth2_client(old_account) + if client is None: + return True + + resp = client.get("https://api.github.com/user") + if resp.status_code == 401: + return True + + return False + + def _has_new_account_for_old_account(self): + """ + Check if the user has connected his account to the new GitHub App. + + The new connected account must the same as the old one. + """ + query = self.request.user.socialaccount_set.filter( + provider=GitHubAppProvider.id, + ) + old_account = self._get_old_github_account() + if old_account: + query.filter(uid=old_account.uid) + return query.exists() + + def _get_new_github_account(self): + return self.request.user.socialaccount_set.filter(provider=GitHubAppProvider.id).first() + + def _get_old_github_account(self): + return self.request.user.socialaccount_set.filter(provider=GitHubProvider.id).first() + + def post(self, request, *args, **kwargs): + project_slug = request.POST.get("project") + if project_slug: + projects = AdminPermission.projects(request.user, admin=True).filter(slug=project_slug) + else: + projects = get_valid_projects_missing_migration(request.user) + + for project in projects: + result = migrate_project_to_github_app(project=project, user=request.user) + if not result.webhook_removed: + Notification.objects.add( + message_id=MESSAGE_OAUTH_WEBHOOK_NOT_REMOVED, + attached_to=request.user, + dismissable=True, + format_values={ + "repo_full_name": project.remote_repository.full_name, + "project_slug": project.slug, + }, + ) + if not result.ssh_key_removed: + Notification.objects.add( + message_id=MESSAGE_OAUTH_DEPLOY_KEY_NOT_REMOVED, + attached_to=request.user, + dismissable=True, + format_values={ + "repo_full_name": project.remote_repository.full_name, + "project_slug": project.slug, + }, + ) + + return HttpResponseRedirect(reverse("migrate_to_github_app") + "?step=migrate") diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py index 3fcad60270a..fd24ba61171 100644 --- a/readthedocs/projects/models.py +++ b/readthedocs/projects/models.py @@ -980,6 +980,7 @@ def vcs_repo(self, environment, version): self, version=version, environment=environment, + token=self.clone_token, ) return repo @@ -1410,6 +1411,24 @@ def get_subproject_candidates(self, user): def organization(self): return self.organizations.first() + @property + def clone_token(self): + """ + Return a token for HTTP Git clone access to the repository. + + .. note:: + + Only repositories granted acces by a GitHub app installation will return a token. + """ + service_class = self.get_git_service_class() + if not service_class: + return None + for service in service_class.for_project(self): + token = service.get_clone_token(self) + if token: + return token + return None + class APIProject(Project): """ @@ -1426,12 +1445,17 @@ class APIProject(Project): """ features = [] + # This is a property in the original model, in order to + # be able to assign it a value in the constructor, we need to re-declare it + # as an attribute here. + clone_token = None class Meta: proxy = True def __init__(self, *args, **kwargs): self.features = kwargs.pop("features", []) + self.clone_token = kwargs.pop("clone_token", None) environment_variables = kwargs.pop("environment_variables", {}) ad_free = not kwargs.pop("show_advertising", True) # These fields only exist on the API return, not on the model, so we'll