Regression Report - Publish Workflow

[jsjoeio]

Table of Contents

• Background
• Impact
  • Timing
  • Root Cause
    • Example
  • Suggested Actions

Background

We recently modified our CI workflow (#2866) in an effort to get CI working on forks, but also improve our CI workflow (using actions/checkout@v2 for example). Our CI and tests passed per usual so we didn't expect any regressions to occur.

Impact

The publish/release workflow broke for npm and docker causing the newest release (3.9.2) of code-server to not be published on the npm/Docker registry.

Timing

Event	Date	Link	Author
Introduced At	2021-03-11	#2866	@jawnsy
Reported At	2020-03-25	#2972 #2973	@jsjoeio @code-asher @aptalca @nodesocket
Resolved At	2020-03-26	#2968 #2974 #2981 690862306	@jsjoeio

Root Cause

Current maintainers approved #2866 without realizing that the release process (upload artifacts and publish to registries) relied on push events (see code here).

When this change was made, it broke the functionality of the get_artifacts_url in the ./ci/lib.sh script. This script is used in 12 scripts used in code-server.

Bug: The release process relied on grabbing the GitHub Actions artifact URL for push events.

Bug: There was no option for triggering a release manually.

1. The user was notified of the new release and expecting it to be available on the npm and Docker registries.
   1. We noticed the issue immediately during the publish workflow but failed to post any announcement to our users via GitHub Issues or Discussions.
      1. We have a practice of noting user-facing issues in Slack instead of GitHub issues or discussions.
2. The release process written by previous maintainers has quirks not understood by current maintainers.
   1. Our release process lacks documentation explaining the nuances behind our process.
   2. We have no tests in place for our release scripts (or any scripts).
      1. Our code reviewers don't scrutinize error handling as much as they should
3. The maintainers had no way to trigger a manual release.
   1. Our publish workflow did not have the workflow action to trigger manually from GitHub's UI.
   2. There is no documentation for manually publishing to npm or Docker.
      1. Not all current maintainers have access to proper admin rights or tokens for publishing on the registries.

Suggested Actions

• Post a GitHub issue or Discussion when an error with the release process occurs

  • Create docs/MAINTAINERS_HANDBOOK.md and document this process Document error communication with community #2991

• Improve the current release process

  • Refactor to be less brittle where possible Refactor release process: attaching artifacts #2995
  • Document scripts used in release process Document scripts used in release process #2992
  • Write tests for our scripts using bats Write tests for dev and build scripts #2994
  • run macos-amd64 job on release PRs [dev]: run macos-amd64 job on release PRs #2970

• Document how to manually release a new version of code-server

  • Ensure all maintainers (and a backup maintainer/Coder employee) have proper rights and access to tokens for publishing on registries Document how to manually release a new version of code-server #2997
  • Use bot tokens for publish (see latest Docker release - says it was pushed by previous maintainer) Replace release tokens (npm, Docker) with bot tokens #2996

  Note: @jsjoeio to create follow-up issues for each of the Suggested Actions Completed on 3/29/2021