feat: Support GitHub Enterprise Cloud with Data Residency (github-aws-runners#4390)

Opening up the PR again to address the CI issue -
github-aws-runners#4389

---------

Co-authored-by: Neethu Pandhaplavil <[email protected]>
Co-authored-by: Jørgen Jervidalo <[email protected]>
Co-authored-by: Niek Palm <[email protected]>

Author: 4 people

README.md

@@ -25,7 +25,7 @@ This [Terraform](https://www.terraform.io/) module creates the required infrastr
 - Tailored software, hardware and network configuration: Bring your own AMI, define the instance types and subnets to use.
 - OS support: Linux (x64/arm64) and Windows
 - Multi-Runner: Create multiple runner configurations with a single deployment
-- GitHub cloud and GitHub Enterprise Server (GHES) support.
+- GitHub cloud, Github Cloud with Data Residency and GitHub Enterprise Server (GHES) support.
 - Org and repo level runners. enterprise level runners are not supported (yet).
 
 
@@ -140,7 +140,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
 | <a name="input_enable_userdata"></a> [enable\_userdata](#input\_enable\_userdata) | Should the userdata script be enabled for the runner. Set this to false if you are using your own prebuilt AMI. | `bool` | `true` | no |
 | <a name="input_eventbridge"></a> [eventbridge](#input\_eventbridge) | Enable the use of EventBridge by the module. By enabling this feature events will be put on the EventBridge by the webhook instead of directly dispatching to queues for scaling.<br/><br/>    `enable`: Enable the EventBridge feature.<br/>    `accept_events`: List can be used to only allow specific events to be putted on the EventBridge. By default all events, empty list will be be interpreted as all events. | <pre>object({<br/>    enable        = optional(bool, true)<br/>    accept_events = optional(list(string), null)<br/>  })</pre> | `{}` | no |
 | <a name="input_ghes_ssl_verify"></a> [ghes\_ssl\_verify](#input\_ghes\_ssl\_verify) | GitHub Enterprise SSL verification. Set to 'false' when custom certificate (chains) is used for GitHub Enterprise Server (insecure). | `bool` | `true` | no |
-| <a name="input_ghes_url"></a> [ghes\_url](#input\_ghes\_url) | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB | `string` | `null` | no |
+| <a name="input_ghes_url"></a> [ghes\_url](#input\_ghes\_url) | GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB - github.com. However if you are using Github Enterprise Cloud with data-residency (ghe.com), set the endpoint here. Example - https://companyname.ghe.com | `string` | `null` | no |
 | <a name="input_github_app"></a> [github\_app](#input\_github\_app) | GitHub app parameters, see your github app. Ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`). | <pre>object({<br/>    key_base64     = string<br/>    id             = string<br/>    webhook_secret = string<br/>  })</pre> | n/a | yes |
 | <a name="input_idle_config"></a> [idle\_config](#input\_idle\_config) | List of time periods, defined as a cron expression, to keep a minimum amount of runners active instead of scaling down to 0. By defining this list you can ensure that in time periods that match the cron expression within 5 seconds a runner is kept idle. | <pre>list(object({<br/>    cron             = string<br/>    timeZone         = string<br/>    idleCount        = number<br/>    evictionStrategy = optional(string, "oldest_first")<br/>  }))</pre> | `[]` | no |
 | <a name="input_instance_allocation_strategy"></a> [instance\_allocation\_strategy](#input\_instance\_allocation\_strategy) | The allocation strategy for spot instances. AWS recommends using `price-capacity-optimized` however the AWS default is `lowest-price`. | `string` | `"lowest-price"` | no |

docs/configuration.md

@@ -10,7 +10,7 @@ To be able to support a number of use-cases, the module has quite a lot of confi
 - Linux vs Windows. You can configure the OS types linux and win. Linux will be used by default.
 - Re-use vs Ephemeral. By default runners are re-used, until detected idle. Once idle they will be removed from the pool. To improve security we are introducing ephemeral runners. Those runners are only used for one job. Ephemeral runners only work in combination with the workflow job event. For ephemeral runners the lambda requests a JIT (just in time) configuration via the GitHub API to register the runner. [JIT configuration](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-just-in-time-runners) is limited to ephemeral runners (and currently not supported by GHES). For non-ephemeral runners, a registration token is always requested. In both cases the configuration is made available to the instance via the same SSM parameter. To disable JIT configuration for ephemeral runners set `enable_jit_config` to `false`. We also suggest using a pre-build AMI to improve the start time of jobs for ephemeral runners.
 - Job retry (**Beta**). By default the scale-up lambda will discard the message when it is handled. Meaning in the ephemeral use-case an instance is created. The created runner will ask GitHub for a job, no guarantee it will run the job for which it was scaling. Result could be that with small system hick-up the job is keeping waiting for a runner. Enable a pool (org runners) is one option to avoid this problem. Another option is to enable the job retry function. Which will retry the job after a delay for a configured number of times.
-- GitHub Cloud vs GitHub Enterprise Server (GHES). The runners support GitHub Cloud as well GitHub Enterprise Server. For GHES, we rely on our community for support and testing. We have no capability to test GHES ourselves.
+- GitHub Cloud vs GitHub Enterprise Server (GHES). The runners support  GitHub Cloud (Public GitHub - github.com), GitHub Data Residency instances (ghe.com), and GitHub Enterprise Server. For GHES, we rely on our community for support and testing. We have no capability to test GHES ourselves.
 - Spot vs on-demand. The runners use either the EC2 spot or on-demand life cycle. Runners will be created via the AWS [CreateFleet API](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html). The module (scale up lambda) will request via the CreateFleet API to create instances in one of the subnets and of the specified instance types.
 - ARM64 support via Graviton/Graviton2 instance-types. When using the default example or top-level module, specifying `instance_types` that match a Graviton/Graviton 2 (ARM64) architecture (e.g. a1, t4g or any 6th-gen `g` or `gd` type), you must also specify `runner_architecture = "arm64"` and the sub-modules will be automatically configured to provision with ARM64 AMIs and leverage GitHub's ARM64 action runner. See below for more details.
 - Disable default labels for the runners (os, architecture and `self-hosted`) can achieve by setting `runner_disable_default_labels` = true. If enabled, the runner will only have the extra labels provided in `runner_extra_labels`. In case you on own start script is used, this configuration parameter needs to be parsed via SSM.

lambdas/functions/control-plane/src/pool/pool.test.ts

@@ -5,7 +5,7 @@ import nock from 'nock';
 
 import { listEC2Runners } from '../aws/runners';
 import * as ghAuth from '../github/auth';
-import { createRunners } from '../scale-runners/scale-up';
+import { createRunners, getGitHubEnterpriseApiUrl } from '../scale-runners/scale-up';
 import { adjust } from './pool';
 
 const mockOctokit = {
@@ -28,7 +28,7 @@ jest.mock('./../aws/runners', () => ({
   listEC2Runners: jest.fn(),
 }));
 jest.mock('./../github/auth');
-jest.mock('./../scale-runners/scale-up');
+jest.mock('../scale-runners/scale-up');
 
 const mocktokit = Octokit as jest.MockedClass<typeof Octokit>;
 const mockedAppAuth = mocked(ghAuth.createGithubAppAuth, {
@@ -167,6 +167,12 @@ beforeEach(() => {
 
 describe('Test simple pool.', () => {
   describe('With GitHub Cloud', () => {
+    beforeEach(() => {
+      (getGitHubEnterpriseApiUrl as jest.Mock).mockReturnValue({
+        ghesApiUrl: '',
+        ghesBaseUrl: '',
+      });
+    });
     it('Top up pool with pool size 2 registered.', async () => {
       await expect(await adjust({ poolSize: 3 })).resolves;
       expect(createRunners).toHaveBeenCalledTimes(1);
@@ -240,7 +246,29 @@ describe('Test simple pool.', () => {
 
   describe('With GHES', () => {
     beforeEach(() => {
-      process.env.GHES_URL = 'https://github.enterprise.something';
+      (getGitHubEnterpriseApiUrl as jest.Mock).mockReturnValue({
+        ghesApiUrl: 'https://api.github.enterprise.something',
+        ghesBaseUrl: 'https://github.enterprise.something',
+      });
+    });
+
+    it('Top up if the pool size is set to 5', async () => {
+      await expect(await adjust({ poolSize: 5 })).resolves;
+      // 2 idle, top up with 3 to match a pool of 5
+      expect(createRunners).toHaveBeenCalledWith(
+        expect.anything(),
+        expect.objectContaining({ numberOfRunners: 3 }),
+        expect.anything(),
+      );
+    });
+  });
+
+  describe('With Github Data Residency', () => {
+    beforeEach(() => {
+      (getGitHubEnterpriseApiUrl as jest.Mock).mockReturnValue({
+        ghesApiUrl: 'https://api.companyname.ghe.com',
+        ghesBaseUrl: 'https://companyname.ghe.com',
+      });
     });
 
     it('Top up if the pool size is set to 5', async () => {

lambdas/functions/control-plane/src/pool/pool.ts

@@ -5,7 +5,7 @@ import yn from 'yn';
 import { bootTimeExceeded, listEC2Runners } from '../aws/runners';
 import { RunnerList } from '../aws/runners.d';
 import { createGithubAppAuth, createGithubInstallationAuth, createOctokitClient } from '../github/auth';
-import { createRunners } from '../scale-runners/scale-up';
+import { createRunners, getGitHubEnterpriseApiUrl } from '../scale-runners/scale-up';
 
 const logger = createChildLogger('pool');
 
@@ -24,7 +24,6 @@ export async function adjust(event: PoolEvent): Promise<void> {
   const runnerGroup = process.env.RUNNER_GROUP_NAME || '';
   const runnerNamePrefix = process.env.RUNNER_NAME_PREFIX || '';
   const environment = process.env.ENVIRONMENT;
-  const ghesBaseUrl = process.env.GHES_URL;
   const ssmTokenPath = process.env.SSM_TOKEN_PATH;
   const ssmConfigPath = process.env.SSM_CONFIG_PATH || '';
   const subnets = process.env.SUBNET_IDS.split(',');
@@ -43,10 +42,7 @@ export async function adjust(event: PoolEvent): Promise<void> {
     ? (JSON.parse(process.env.ENABLE_ON_DEMAND_FAILOVER_FOR_ERRORS) as [string])
     : [];
 
-  let ghesApiUrl = '';
-  if (ghesBaseUrl) {
-    ghesApiUrl = `${ghesBaseUrl}/api/v3`;
-  }
+  const { ghesApiUrl, ghesBaseUrl } = getGitHubEnterpriseApiUrl();
 
   const installationId = await getInstallationId(ghesApiUrl, runnerOwner);
   const ghAuth = await createGithubInstallationAuth(installationId, ghesApiUrl);

lambdas/functions/control-plane/src/scale-runners/job-retry.ts

@@ -1,6 +1,6 @@
 import { addPersistentContextToChildLogger, createSingleMetric, logger } from '@aws-github-runner/aws-powertools-util';
 import { publishMessage } from '../aws/sqs';
-import { ActionRequestMessage, ActionRequestMessageRetry, getGitHubEnterpriseApiUrl, isJobQueued } from './scale-up';
+import { ActionRequestMessage, ActionRequestMessageRetry, isJobQueued, getGitHubEnterpriseApiUrl } from './scale-up';
 import { getOctokit } from '../github/octokit';
 import { MetricUnit } from '@aws-lambda-powertools/metrics';
 import yn from 'yn';

lambdas/functions/control-plane/src/scale-runners/scale-down.test.ts

@@ -159,11 +159,11 @@ describe('Scale down runners', () => {
     mockCreateClient.mockResolvedValue(new mocktokit());
   });
 
-  const endpoints = ['https://api.github.com', 'https://github.enterprise.something'];
+  const endpoints = ['https://api.github.com', 'https://github.enterprise.something', 'https://companyname.ghe.com'];
 
   describe.each(endpoints)('for %s', (endpoint) => {
     beforeEach(() => {
-      if (endpoint.includes('enterprise')) {
+      if (endpoint.includes('enterprise') || endpoint.endsWith('.ghe.com')) {
         process.env.GHES_URL = endpoint;
       }
     });

lambdas/functions/control-plane/src/scale-runners/scale-down.ts

@@ -8,6 +8,7 @@ import { RunnerInfo, RunnerList } from './../aws/runners.d';
 import { GhRunners, githubCache } from './cache';
 import { ScalingDownConfig, getEvictionStrategy, getIdleRunnerCount } from './scale-down-config';
 import { metricGitHubAppRateLimit } from '../github/rate-limit';
+import { getGitHubEnterpriseApiUrl } from './scale-up';
 
 const logger = createChildLogger('scale-down');
 
@@ -21,11 +22,7 @@ async function getOrCreateOctokit(runner: RunnerInfo): Promise<Octokit> {
   }
 
   logger.debug(`[createGitHubClientForRunner] Cache miss for ${key}`);
-  const ghesBaseUrl = process.env.GHES_URL;
-  let ghesApiUrl = '';
-  if (ghesBaseUrl) {
-    ghesApiUrl = `${ghesBaseUrl}/api/v3`;
-  }
+  const { ghesApiUrl } = getGitHubEnterpriseApiUrl();
   const ghAuthPre = await createGithubAppAuth(undefined, ghesApiUrl);
   const githubClientPre = await createOctokitClient(ghAuthPre.token, ghesApiUrl);