action, twine-upload: add dry_run setting

woodruffw · woodruffw · commit 0d8546436bea · 2023-03-13T12:17:56.000-04:00
Just for testing. Signed-off-by: William Woodruff <william@trailofbits.com> action, twine-upload: scaffolding Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: initial skeleton Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange, twine-upload: tweakage Signed-off-by: William Woodruff <william@trailofbits.com> twine, oidc: move mask back into exchange Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: TestPyPI support Signed-off-by: William Woodruff <william@trailofbits.com> action: remove oidc_audience input Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: debugging Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: debugging Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: typo Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: audience negotiation Signed-off-by: William Woodruff <william@trailofbits.com> debug: dump JWT payload This is not sensitive, since we strip the signature. Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: typo Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: debugging Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: oopsie Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: undo debugging changes Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: switch to `id` for OIDC cred Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: better error messages/step summaries Signed-off-by: William Woodruff <william@trailofbits.com> remove `dry_run` Signed-off-by: William Woodruff <william@trailofbits.com> requirements: `pip-compile` Signed-off-by: William Woodruff <william@trailofbits.com> Bump cryptography from 38.0.4 to 39.0.1 in /requirements Bumps [cryptography](https://github.com/pyca/cryptography) from 38.0.4 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@38.0.4...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> ⇪ Bump isort to v5.12.0 The previous version had a Poetry packaging problem. This patch fixes that. 🎨 Warn about empty password/token action input Before this patch, the warning would say that the token was expected to start with `pypi-` but it may be unobvious. With this change, the end-users are warned when they're passing a completely empty password value. Fixes #25. 🎨 Convert action inputs to use kebab-case Up until now, the action input names followed the snake_case naming pattern that is well familiar to the pythonistas. But in GitHub actions, the de-facto standard is using kebab-case, which is what this patch achieves. This style helps make the keys in YAML better standardized and distinguishable from other identifiers. The old snake_case names remain functional for the time being and will not be removed until at least v3 release of this action. 🐛 Make kebab options fall back for snake_case The previous release didn't take into account the action defaults so the promised fallbacks for the old input names didn't work. This patch corrects that mistake. oidc-exchange: context manager Signed-off-by: William Woodruff <william@trailofbits.com> twine-upload: reflow Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: input normalization Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: reflow Signed-off-by: William Woodruff <william@trailofbits.com> runtime.in: document dependency Signed-off-by: William Woodruff <william@trailofbits.com> twine-upload: enquote Signed-off-by: William Woodruff <william@trailofbits.com> twine-upload: only do OIDC flow when user is token Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: reflow Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: reflow Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exhcange: factor out audience call check Signed-off-by: William Woodruff <william@trailofbits.com> README: document OIDC publishing Signed-off-by: William Woodruff <william@trailofbits.com> oidc-exchange: reflow Signed-off-by: William Woodruff <william@trailofbits.com>
diff --git a/.github/workflows/self-smoke-test-action.yml b/.github/workflows/self-smoke-test-action.yml
@@ -91,7 +91,7 @@ jobs:
       with:
         user: ${{ env.devpi-username }}
         password: ${{ env.devpi-password }}
-        repository_url: >-
+        repository-url: >-
           http://devpi:${{ env.devpi-port }}/${{ env.devpi-username }}/public/
 
 ...
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -9,7 +9,7 @@ repos:
   - id: add-trailing-comma
 
 - repo: https://github.com/PyCQA/isort.git
-  rev: 5.11.4
+  rev: 5.12.0
   hooks:
   - id: isort
     args:
diff --git a/Dockerfile b/Dockerfile
@@ -25,6 +25,7 @@ WORKDIR /app
 COPY LICENSE.md .
 COPY twine-upload.sh .
 COPY print-hash.py .
+COPY oidc-exchange.py .
 
 RUN chmod +x twine-upload.sh
 ENTRYPOINT ["/app/twine-upload.sh"]
diff --git a/README.md b/README.md
@@ -61,6 +61,46 @@ PyPI, which is recommended to restrict the access the action has.
 The secret used in `${{ secrets.PYPI_API_TOKEN }}` needs to be created on the
 settings page of your project on GitHub. See [Creating & using secrets].
 
+### Publishing with OpenID Connect
+
+**IMPORTANT**: This functionality is in beta, and will not work for you
+unless you're a member of the PyPI OIDC beta testers' group. For more
+information, see
+[warehouse#12965](https://github.com/pypi/warehouse/issues/12965).
+
+This action supports PyPI's
+[OpenID Connect publishing](https://pypi.org/help/#openid-connect)
+implementation, which allows authentication to PyPI without a manually
+configured API token or username/password combination. To perform
+OIDC publishing with this action, your project's OIDC publisher must
+already be configured on PyPI.
+
+To enter the OIDC flow, configure this action's job with the `id-token: write`
+permission and **without** an explicit username or password:
+
+```yaml
+jobs:
+  pypi-publish:
+    name: upload release to PyPI
+    runs-on: ubuntu-latest
+    permissions:
+      # IMPORTANT: this permission is mandatory for OIDC publishing
+      id-token: write
+    steps:
+      # retrieve your distributions here
+
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+```
+
+Other indices that support OIDC publishing can also be used, like TestPyPI:
+
+```yaml
+- name: Publish package distributions to TestPyPI
+  uses: pypa/gh-action-pypi-publish@release/v1
+  with:
+    repository-url: https://test.pypi.org/legacy/
+```
 
 ## Non-goals
 
@@ -99,7 +139,7 @@ project's specific needs.
 For example, you could implement a parallel workflow that
 pushes every commit to TestPyPI or your own index server,
 like `devpi`. For this, you'd need to (1) specify a custom
-`repository_url` value and (2) generate a unique version
+`repository-url` value and (2) generate a unique version
 number for each upload so that they'd not create a conflict.
 The latter is possible if you use `setuptools_scm` package but
 you could also invent your own solution based on the distance
@@ -114,7 +154,7 @@ The action invocation in this case would look like:
   uses: pypa/gh-action-pypi-publish@release/v1
   with:
     password: ${{ secrets.TEST_PYPI_API_TOKEN }}
-    repository_url: https://test.pypi.org/legacy/
+    repository-url: https://test.pypi.org/legacy/
 ```
 
 ### Customizing target package dists directory
@@ -128,7 +168,7 @@ would now look like:
   uses: pypa/gh-action-pypi-publish@release/v1
   with:
     password: ${{ secrets.PYPI_API_TOKEN }}
-    packages_dir: custom-dir/
+    packages-dir: custom-dir/
 ```
 
 ### Disabling metadata verification
@@ -139,7 +179,7 @@ check with:
 
 ```yml
    with:
-     verify_metadata: false
+     verify-metadata: false
 ```
 
 ### Tolerating release package file duplicates
@@ -149,12 +189,12 @@ may hit race conditions. For example, when publishing from multiple CIs
 or even having workflows with the same steps triggered within GitHub
 Actions CI/CD for different events concerning the same high-level act.
 
-To facilitate this use-case, you may use `skip_existing` (disabled by
+To facilitate this use-case, you may use `skip-existing` (disabled by
 default) setting as follows:
 
 ```yml
    with:
-     skip_existing: true
+     skip-existing: true
 ```
 
 > **Pro tip**: try to avoid enabling this setting where possible. If you
@@ -177,7 +217,7 @@ It will show SHA256, MD5, BLAKE2-256 values of files to be uploaded.
 
 ```yml
   with:
-    print_hash: true
+    print-hash: true
 ```
 
 ### Specifying a different username
diff --git a/action.yml b/action.yml
@@ -9,30 +9,75 @@ inputs:
   password:
     description: Password for your PyPI user or an access token
     required: true
-  repository_url:
+  repository-url:  # Canonical alias for `repository_url`
     description: The repository URL to use
     required: false
-  packages_dir:
+  repository_url:  # DEPRECATED ALIAS; TODO: Remove in v3+
+    description: >-
+      [DEPRECATED]
+      The repository URL to use
+    deprecationMessage: >-
+      The inputs have been normalized to use kebab-case.
+      Use `repository-url` instead.
+    required: false
+  packages-dir:  # Canonical alias for `packages_dir`
     description: The target directory for distribution
     required: false
+    # default: dist  # TODO: uncomment once alias removed
+  packages_dir:  # DEPRECATED ALIAS; TODO: Remove in v3+
+    description: >-
+      [DEPRECATED]
+      The target directory for distribution
+    deprecationMessage: >-
+      The inputs have been normalized to use kebab-case.
+      Use `packages-dir` instead.
+    required: false
     default: dist
-  verify_metadata:
+  verify-metadata:  # Canonical alias for `verify_metadata`
     description: Check metadata before uploading
     required: false
+    # default: 'true'  # TODO: uncomment once alias removed
+  verify_metadata:  # DEPRECATED ALIAS; TODO: Remove in v3+
+    description: >-
+      [DEPRECATED]
+      Check metadata before uploading
+    deprecationMessage: >-
+      The inputs have been normalized to use kebab-case.
+      Use `verify-metadata` instead.
+    required: false
     default: 'true'
-  skip_existing:
+  skip-existing:  # Canonical alias for `skip_existing`
     description: >-
       Do not fail if a Python package distribution
       exists in the target package index
     required: false
+    # default: 'false'  # TODO: uncomment once alias removed
+  skip_existing:  # DEPRECATED ALIAS; TODO: Remove in v3+
+    description: >-
+      [DEPRECATED]
+      Do not fail if a Python package distribution
+      exists in the target package index
+    deprecationMessage: >-
+      The inputs have been normalized to use kebab-case.
+      Use `skip-existing` instead.
+    required: false
     default: 'false'
   verbose:
     description: Show verbose output.
     required: false
     default: 'false'
-  print_hash:
+  print-hash:  # Canonical alias for `print_hash`
     description: Show hash values of files to be uploaded
     required: false
+    # default: 'false'  # TODO: uncomment once alias removed
+  print_hash:  # DEPRECATED ALIAS; TODO: Remove in v3+
+    description: >-
+      [DEPRECATED]
+      Show hash values of files to be uploaded
+    deprecationMessage: >-
+      The inputs have been normalized to use kebab-case.
+      Use `print-hash` instead.
+    required: false
     default: 'false'
 branding:
   color: yellow
@@ -43,9 +88,9 @@ runs:
   args:
   - ${{ inputs.user }}
   - ${{ inputs.password }}
-  - ${{ inputs.repository_url }}
-  - ${{ inputs.packages_dir }}
-  - ${{ inputs.verify_metadata }}
-  - ${{ inputs.skip_existing }}
+  - ${{ inputs.repository-url }}
+  - ${{ inputs.packages-dir }}
+  - ${{ inputs.verify-metadata }}
+  - ${{ inputs.skip-existing }}
   - ${{ inputs.verbose }}
-  - ${{ inputs.print_hash }}
+  - ${{ inputs.print-hash }}
diff --git a/oidc-exchange.py b/oidc-exchange.py
@@ -0,0 +1,156 @@
+import os
+from pathlib import Path
+import sys
+from textwrap import dedent
+from typing import NoReturn
+from urllib.parse import urlparse
+
+import id
+import requests
+
+_GITHUB_STEP_SUMMARY = Path(os.getenv("GITHUB_STEP_SUMMARY"))
+
+_TOKEN_RETRIEVAL_FAILED_MESSAGE = dedent(
+    """
+    OIDC token retrieval failed: {identity_error}
+
+    This generally indicates a workflow configuration error, such as insufficient
+    permissions. Make sure that your workflow has `id-token: write` configured
+    at either the workflow or job level, e.g.:
+
+    ```yaml
+    permissions:
+      id-token: write
+    ```
+    """
+)
+
+
+def die(msg: str) -> NoReturn:
+    with _GITHUB_STEP_SUMMARY.open("a") as io:
+        print(msg, file=io)
+
+    # NOTE: `msg` is Markdown formatted, so we emit only the header line to
+    # avoid clogging the console log with a full Markdown formatted document.
+    print(f"::error::OIDC exchange failure: {msg.splitlines()[0]}", file=sys.stderr)
+    sys.exit(1)
+
+
+def debug(msg: str):
+    print(f"::debug::{msg}", file=sys.stderr)
+
+
+def get_normalized_input(name: str) -> str | None:
+    name = f"INPUT_{name.upper()}"
+    if val := os.getenv(name):
+        return val
+    return os.getenv(name.replace("-", "_"))
+
+
+def assert_successful_audience_call(resp: requests.Response, domain: str):
+    if resp.ok:
+        return
+
+    match resp.status_code:
+        case 403:
+            # This index supports OIDC, but forbids the client from using
+            # it (either because it's disabled, ratelimited, etc.)
+            die(f"audience retrieval failed: repository at {domain} has OIDC disabled")
+        case 404:
+            # This index does not support OIDC.
+            die(
+                "audience retrieval failed: repository at "
+                f"{domain} does not indicate OIDC support"
+            )
+        case other:
+            # Unknown: the index may or may not support OIDC, but didn't respond with
+            # something we expect. This can happen if the index is broken, in maintenance mode,
+            # misconfigured, etc.
+            die(
+                "audience retrieval failed: repository at "
+                f"{domain} responded with unexpected {other}"
+            )
+
+
+repository_url = get_normalized_input("repository-url")
+if not repository_url:
+    # Easy case: no explicit repository URL, which means we're using PyPI and we can just
+    # hardcode the exchange endpoint and OIDC audience.
+    token_exchange_url = "https://pypi.org/_/oidc/github/mint-token"
+    oidc_audience = "pypi"
+else:
+    repository_domain = urlparse(repository_url).netloc
+    token_exchange_url = f"https://{repository_domain}/_/oidc/github/mint-token"
+
+    # Indices are expected to support `https://{domain}/_/oidc/audience`,
+    # which tells OIDC exchange clients which audience to use.
+    audience_url = f"https://{repository_domain}/_/oidc/audience"
+    audience_resp = requests.get(audience_url)
+    assert_successful_audience_call(audience_resp, repository_domain)
+
+    oidc_audience = audience_resp.json()["audience"]
+
+debug(f"selected OIDC token exchange endpoint: {token_exchange_url}")
+
+try:
+    oidc_token = id.detect_credential(audience=oidc_audience)
+except id.IdentityError as identity_error:
+    die(_TOKEN_RETRIEVAL_FAILED_MESSAGE.format(identity_error=identity_error))
+
+# Now we can do the actual token exchange.
+mint_token_resp = requests.post(
+    token_exchange_url,
+    json={"token": oidc_token},
+)
+
+if not mint_token_resp.ok:
+    try:
+        error_payload = mint_token_resp.json()
+    except requests.JSONDecodeError:
+        # Token exchange failure normally produces a JSON error response, but
+        # we might have hit a server error instead.
+        die(
+            dedent(
+                f"""
+                Token request failed: the index produced an unexpected
+                {mint_token_resp.status_code} response.
+
+                This strongly suggests a server configuration or downtime issue; wait
+                a few minutes and try again.
+                """
+            )
+        )
+
+    reasons = "\n".join(
+        f"* `{error['code']}`: {error['description']}"
+        for error in error_payload["errors"]
+    )
+
+    # NOTE: Can't `dedent(...)` here because `reasons` is newline-delimited.
+    die(
+        f"""
+Token request failed: the server refused the request for the following reasons:
+
+{reasons}
+        """
+    )
+
+mint_token_payload = mint_token_resp.json()
+pypi_token = mint_token_payload.get("token")
+if pypi_token is None:
+    die(
+        dedent(
+            """
+            Token response error: the index gave us an invalid response.
+
+            This strongly suggests a server configuration or downtime issue; wait
+            a few minutes and try again.
+            """
+        )
+    )
+
+# Mask the newly minted PyPI token, so that we don't accidentally leak it in logs.
+print(f"::add-mask::{pypi_token}", file=sys.stderr)
+
+# This final print will be captured by the subshell in `twine-upload.sh`.
+print(pypi_token)
diff --git a/requirements/runtime.in b/requirements/runtime.in
@@ -1,5 +1,13 @@
 twine
 
+# NOTE: Used to detect an ambient OIDC credential for OIDC publishing.
+id ~= 1.0
+
+# NOTE: This is pulled in transitively through `twine`, but we also declare
+# NOTE: it explicitly here because `oidc-exchange.py` uses it.
+# Ref: https://github.com/di/id
+requests
+
 # NOTE: `pkginfo` is a transitive dependency for us that is coming from Twine.
 # NOTE: It is declared here only to avoid installing a broken combination of
 # NOTE: the distribution packages. This should be removed once a fixed version
diff --git a/requirements/runtime.txt b/requirements/runtime.txt
diff --git a/twine-upload.sh b/twine-upload.sh
