Skip to content
This repository was archived by the owner on Jan 16, 2025. It is now read-only.

Commit c08b335

Browse files
GuptaNavdeep1983npalmnavdeepg2021
authored
feat: Add multi-runner capability (#2472)
* feat: Remove support check_run (#2521) * chore: Remove support check_run * format, lint * feat: Remove old scale down mechanism (< 0.19.0) (#2519) fix: Remove old cleanup mechanism (< 0.19.0) * feat: added changes for multi runner. * fix: region. * fix: more fixes. * tuple to list. * fixes. * fixes. * fixes. * fixes. * fixes. * fixes. * fix: formatting. * fix: formatting. * fix: formatting. * fix: moved some blocks outside runner config. * fix: few more updates * fix: liniting. * fix: updated example output * changed runner group name. * fix: updated the tests. * fix: addressed review comments. * fix: linting issues. * fix: formatting. * fix: updated tf version. * fix: Remove removed prerelease option * Add ubuntu runner to example * refactor: use each instead of count * fix: few small issues. * refactor: syncer to count for multi runner * fix: comments. * fix: added Readme. * fix: errors. * move variable to runner config * fix: updated the readme. * Add todos * feat: added windows runner configuration, completed todos and added the weight for runner config matchers. * chore: Update docs * fix: reverted tf versions. * fix: addressed comments. * fix: missed. * fix: formatting. * Update terraform versions in CI * Update terraform versions in CI * Update docs * fix: coverage. * Update docs * improve test coverage webhook * Apply suggestions from code review * fix: formatting. * fix: fixed merge issues. * fix: syntax. Co-authored-by: Niek Palm <[email protected]> Co-authored-by: Niek Palm <[email protected]> Co-authored-by: navdeepg2021 <[email protected]>
1 parent a001003 commit c08b335

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+1827
-283
lines changed

Diff for: .github/workflows/terraform.yml

+31-4
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ jobs:
1515
name: Verify module
1616
strategy:
1717
matrix:
18-
terraform: [1.1.3, "latest"]
18+
terraform: [1.3.2, "latest"]
1919
runs-on: ubuntu-latest
2020
container:
2121
image: hashicorp/terraform:${{ matrix.terraform }}
@@ -29,7 +29,7 @@ jobs:
2929
touch modules/runner-binaries-syncer/lambdas/runner-binaries-syncer/runner-binaries-syncer.zip
3030
- name: terraform init
3131
run: terraform init -get -backend=false -input=false
32-
- if: contains(matrix.terraform, '1.1.')
32+
- if: contains(matrix.terraform, '1.3.')
3333
name: check terraform formatting
3434
run: terraform fmt -recursive -check=true -write=false
3535
- if: contains(matrix.terraform, 'latest') # check formatting for the latest release but avoid failing the build
@@ -44,7 +44,7 @@ jobs:
4444
strategy:
4545
fail-fast: false
4646
matrix:
47-
terraform: [1.0.11, 1.1.3, "latest"]
47+
terraform: [1.0.11, 1.1.9, 1.2.9, "latest"]
4848
example:
4949
["default", "ubuntu", "prebuilt", "arm64", "ephemeral", "windows"]
5050
defaults:
@@ -57,7 +57,7 @@ jobs:
5757
- uses: actions/checkout@v3
5858
- name: terraform init
5959
run: terraform init -get -backend=false -input=false
60-
- if: contains(matrix.terraform, '1.1.')
60+
- if: contains(matrix.terraform, '1.3.')
6161
name: check terraform formatting
6262
run: terraform fmt -recursive -check=true -write=false
6363
- if: contains(matrix.terraform, 'latest') # check formatting for the latest release but avoid failing the build
@@ -66,3 +66,30 @@ jobs:
6666
continue-on-error: true
6767
- name: validate terraform011
6868
run: terraform validate
69+
70+
71+
verify_multi_runner_example:
72+
name: Verify Multi-Runner examples
73+
strategy:
74+
fail-fast: false
75+
matrix:
76+
terraform: [1.3.2, "latest"]
77+
defaults:
78+
run:
79+
working-directory: examples/multi-runner
80+
runs-on: ubuntu-latest
81+
container:
82+
image: hashicorp/terraform:${{ matrix.terraform }}
83+
steps:
84+
- uses: actions/checkout@v3
85+
- name: terraform init
86+
run: terraform init -get -backend=false -input=false
87+
- if: contains(matrix.terraform, '1.3.')
88+
name: check terraform formatting
89+
run: terraform fmt -recursive -check=true -write=false
90+
- if: contains(matrix.terraform, 'latest') # check formatting for the latest release but avoid failing the build
91+
name: check terraform formatting
92+
run: terraform fmt -recursive -check=true -write=false
93+
continue-on-error: true
94+
- name: validate terraform
95+
run: terraform validate

Diff for: README.md

+11-16
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ This [Terraform](https://www.terraform.io/) module creates the required infrastr
99
- [Motivation](#motivation)
1010
- [Overview](#overview)
1111
- [Major configuration options.](#major-configuration-options)
12-
- [ARM64 support via Graviton/Graviton2 instance-types](#arm64-support-via-gravitongraviton2-instance-types)
1312
- [Usages](#usages)
1413
- [Setup GitHub App (part 1)](#setup-github-app-part-1)
1514
- [Setup terraform module](#setup-terraform-module)
@@ -25,7 +24,6 @@ This [Terraform](https://www.terraform.io/) module creates the required infrastr
2524
- [Experimental - Optional queue to publish GitHub workflow job events](#experimental---optional-queue-to-publish-github-workflow-job-events)
2625
- [Examples](#examples)
2726
- [Sub modules](#sub-modules)
28-
- [ARM64 configuration for submodules](#arm64-configuration-for-submodules)
2927
- [Debugging](#debugging)
3028
- [Security Consideration](#security-consideration)
3129
- [Requirements](#requirements)
@@ -81,16 +79,13 @@ Besides these permissions, the lambdas also need permission to CloudWatch (for l
8179
To be able to support a number of use-cases the module has quite a lot of configuration options. We try to choose reasonable defaults. The several examples also show for the main cases how to configure the runners.
8280

8381
- Org vs Repo level. You can configure the module to connect the runners in GitHub on an org level and share the runners in your org. Or set the runners on repo level and the module will install the runner to the repo. There can be multiple repos but runners are not shared between repos.
84-
- Checkrun vs Workflow job event. You can configure the webhook in GitHub to send checkrun or workflow job events to the webhook. Workflow job events are introduced by GitHub in September 2021 and are designed to support scalable runners. We advise when possible using the workflow job event, you can set `runner_enable_workflow_job_labels_check = true` to let the webhook only accept jobs based on the labels configured. The webhook will check the custom labels provided via the variable `runner_extra_labels` and the GitHub managed labels, "self-hosted", OS and architecture. The OS and architecture are derived from the settings. By default the check is disabled.
82+
- Multi-Runner module. This modules allows to create multiple runner configurations with a single webhook and single GitHub App to simply deployment of different types of runners. Refer to the [ReadMe](.modules/../modules/multi-runner/README.md) for more information to understand the functionality.
83+
- Workflow job event. You can configure the webhook in GitHub to send workflow job events to the webhook. Workflow job events are introduced by GitHub in September 2021 and are designed to support scalable runners. We advise when possible using the workflow job event.
8584
- Linux vs Windows. you can configure the OS types linux and win. Linux will be used by default.
8685
- Re-use vs Ephemeral. By default runners are re-used for till detected idle. Once idle they will be removed from the pool. To improve security we are introducing ephemeral runners. Those runners are only used for one job. Ephemeral runners are only working in combination with the workflow job event. We also suggest using a pre-build AMI to improve the start time of jobs.
8786
- GitHub Cloud vs GitHub Enterprise Server (GHES). The runner support GitHub Cloud as well GitHub Enterprise Server. For GHES we rely on our community to test and support. We have no possibility to test ourselves on GHES.
8887
- Spot vs on-demand. The runners use either the EC2 spot or on-demand life cycle. Runners will be created via the AWS [CreateFleet API](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html). The module (scale up lambda) will request via the CreateFleet API to create instances in one of the subnets and of the specified instance types.
89-
90-
91-
#### ARM64 support via Graviton/Graviton2 instance-types
92-
93-
When using the default example or top-level module, specifying `instance_types` that match a Graviton/Graviton 2 (ARM64) architecture (e.g. a1, t4g or any 6th-gen `g` or `gd` type), you must also specify `runner_architecture = "arm64"` and the sub-modules will be automatically configured to provision with ARM64 AMIs and leverage GitHub's ARM64 action runner. See below for more details.
88+
- ARM64 support via Graviton/Graviton2 instance-types. When using the default example or top-level module, specifying `instance_types` that match a Graviton/Graviton 2 (ARM64) architecture (e.g. a1, t4g or any 6th-gen `g` or `gd` type), you must also specify `runner_architecture = "arm64"` and the sub-modules will be automatically configured to provision with ARM64 AMIs and leverage GitHub's ARM64 action runner. See below for more details.
9489

9590
## Usages
9691

@@ -334,11 +329,13 @@ Examples are located in the [examples](./examples) directory. The following exam
334329

335330
- _[Default](examples/default/README.md)_: The default example of the module
336331
- _[ARM64](examples/arm64/README.md)_: Example usage with ARM64 architecture
337-
- _[Ubuntu](examples/ubuntu/README.md)_: Example usage of creating a runner using Ubuntu AMIs.
338-
- _[Windows](examples/windows/README.md)_: Example usage of creating a runner using Windows as the OS.
339332
- _[Ephemeral](examples/ephemeral/README.md)_: Example usages of ephemeral runners based on the default example.
340-
- _[Prebuilt Images](examples/prebuilt/README.md)_: Example usages of deploying runners with a custom prebuilt image.
333+
- _[Multi Runner](examples/multi-runner/README.md)_ : Example usage of creating a multi runner which creates multiple runners/ configurations with a single deployment
341334
- _[Permissions boundary](examples/permissions-boundary/README.md)_: Example usages of permissions boundaries.
335+
- _[Prebuilt Images](examples/prebuilt/README.md)_: Example usages of deploying runners with a custom prebuilt image.
336+
- _[Ubuntu](examples/ubuntu/README.md)_: Example usage of creating a runner using Ubuntu AMIs.
337+
- _[Windows](examples/windows/README.md)_: Example usage of creating a runner using Windows as the OS.
338+
342339

343340
## Sub modules
344341

@@ -349,15 +346,14 @@ The following submodules are the core of the module and are mandatory:
349346
- _[runner-binaries-syncer](./modules/runner-binaries-syncer/README.md)_ - Syncs the action runner distribution.
350347
- _[runners](./modules/runners/README.md)_ - Scales the action runners up and down
351348
- _[webhook](./modules/webhook/README.md)_ - Handles GitHub webhooks
349+
- _[multi-runner](./modules/multi-runner/README.md) - Creates multiple runner configurations in a single deployment
352350

353351
The following sub modules are optional and are provided as example or utility:
354352

355353
- _[download-lambda](./modules/download-lambda/README.md)_ - Utility module to download lambda artifacts from GitHub Release
356354
- _[setup-iam-permissions](./modules/setup-iam-permissions/README.md)_ - Example module to setup permission boundaries
357355

358-
### ARM64 configuration for submodules
359-
360-
When using the top level module configure `runner_architecture = "arm64"` and ensure the list of `instance_types` matches. When not using the top-level, ensure these properties are set on the submodules.
356+
ARM64 configuration for submodules. When using the top level module configure `runner_architecture = "arm64"` and ensure the list of `instance_types` matches. When not using the top-level, ensure these properties are set on the submodules.
361357

362358
## Debugging
363359

@@ -484,8 +480,7 @@ We welcome any improvement to the standard module to make the default as secure
484480
| <a name="input_runner_boot_time_in_minutes"></a> [runner\_boot\_time\_in\_minutes](#input\_runner\_boot\_time\_in\_minutes) | The minimum time for an EC2 runner to boot and register as a runner. | `number` | `5` | no |
485481
| <a name="input_runner_ec2_tags"></a> [runner\_ec2\_tags](#input\_runner\_ec2\_tags) | Map of tags that will be added to the launch template instance tag specifications. | `map(string)` | `{}` | no |
486482
| <a name="input_runner_egress_rules"></a> [runner\_egress\_rules](#input\_runner\_egress\_rules) | List of egress rules for the GitHub runner instances. | <pre>list(object({<br> cidr_blocks = list(string)<br> ipv6_cidr_blocks = list(string)<br> prefix_list_ids = list(string)<br> from_port = number<br> protocol = string<br> security_groups = list(string)<br> self = bool<br> to_port = number<br> description = string<br> }))</pre> | <pre>[<br> {<br> "cidr_blocks": [<br> "0.0.0.0/0"<br> ],<br> "description": null,<br> "from_port": 0,<br> "ipv6_cidr_blocks": [<br> "::/0"<br> ],<br> "prefix_list_ids": null,<br> "protocol": "-1",<br> "security_groups": null,<br> "self": null,<br> "to_port": 0<br> }<br>]</pre> | no |
487-
| <a name="input_runner_enable_workflow_job_labels_check"></a> [runner\_enable\_workflow\_job\_labels\_check](#input\_runner\_enable\_workflow\_job\_labels\_check) | If set to true all labels in the workflow job even are matched against the custom labels and GitHub labels (os, architecture and `self-hosted`). When the labels are not matching the event is dropped at the webhook. | `bool` | `false` | no |
488-
| <a name="input_runner_enable_workflow_job_labels_check_all"></a> [runner\_enable\_workflow\_job\_labels\_check\_all](#input\_runner\_enable\_workflow\_job\_labels\_check\_all) | If set to true all labels in the workflow job must match the GitHub labels (os, architecture and `self-hosted`). When false if __any__ label matches it will trigger the webhook. `runner_enable_workflow_job_labels_check` must be true for this to take effect. | `bool` | `true` | no |
483+
| <a name="input_runner_enable_workflow_job_labels_check_all"></a> [runner\_enable\_workflow\_job\_labels\_check\_all](#input\_runner\_enable\_workflow\_job\_labels\_check\_all) | If set to true all labels in the workflow job must match the GitHub labels (os, architecture and `self-hosted`). When false if __any__ label matches it will trigger the webhook. | `bool` | `true` | no |
489484
| <a name="input_runner_extra_labels"></a> [runner\_extra\_labels](#input\_runner\_extra\_labels) | Extra (custom) labels for the runners (GitHub). Separate each label by a comma. Labels checks on the webhook can be enforced by setting `enable_workflow_job_labels_check`. GitHub read-only labels should not be provided. | `string` | `""` | no |
490485
| <a name="input_runner_group_name"></a> [runner\_group\_name](#input\_runner\_group\_name) | Name of the runner group. | `string` | `"Default"` | no |
491486
| <a name="input_runner_iam_role_managed_policy_arns"></a> [runner\_iam\_role\_managed\_policy\_arns](#input\_runner\_iam\_role\_managed\_policy\_arns) | Attach AWS or customer-managed IAM policies (by ARN) to the runner IAM role | `list(string)` | `[]` | no |

Diff for: examples/ephemeral/main.tf

-3
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,6 @@ module "runners" {
4343
enable_organization_runners = true
4444
runner_extra_labels = "default,example"
4545

46-
# enable workflow labels check
47-
# runner_enable_workflow_job_labels_check = true
48-
4946
# enable access to the runners via SSM
5047
enable_ssm_on_runners = true
5148

Diff for: examples/multi-runner/.terraform.lock.hcl

+60
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: examples/multi-runner/README.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Action runners deployment of Multiple-Runner-Configurations-Together example
2+
3+
This module shows how to create GitHub action runners with multiple runner configuration together in one deployment. This example has the configurations for the following runner types with the relevant labels supported by them as matchers:
4+
5+
- Linux ARM64 `["self-hosted", "linux", "arm64", "amazon"]`
6+
- Linux Ubuntu `["self-hosted", "linux", "x64", "ubuntu"]`
7+
- Linux X64 `["self-hosted", "linux", "x64", "amazon"]`
8+
- Windows X64 `["self-hosted", "windows", "x64", "servercore-2022"]`
9+
10+
The module will decide the runner for the workflow job based on the match in the labels defined in the workflow job and runner configuration. Also the runner configuration allows the match to be exact or non-exact match. We recommend to use only exact matches.
11+
12+
For exact match, all the labels defined in the workflow should be present in the runner configuration matchers and for non-exact match, some of the labels in the workflow, when present in runner configuration, shall be enough for the runner configuration to be used for the job. First the exact matchers are applied, next the non exact ones.
13+
14+
## Webhook
15+
16+
For the list of provided runner configurations, there will be a single webhook and only a single Github App to receive the notifications for all types of workflow triggers.
17+
18+
## Lambda distribution
19+
20+
Per combination of OS and architecture a lambda distribution syncer will be created. For this example there will be three instances (windows X64, linux X64, linux ARM).
21+
22+
## Usages
23+
24+
Steps for the full setup, such as creating a GitHub app can be found in the root module's [README](../../README.md). First download the Lambda releases from GitHub. Alternatively you can build the lambdas locally with Node or Docker, there is a simple build script in `<root>/.ci/build.sh`. In the `main.tf` you can simply remove the location of the lambda zip files, the default location will work in this case.
25+
26+
> Ensure you have set the version in `lambdas-download/main.tf` for running the example. The version needs to be set to a GitHub release version, see https://github.com/philips-labs/terraform-aws-github-runner/releases
27+
28+
```bash
29+
cd lambdas-download
30+
terraform init
31+
terraform apply
32+
cd ..
33+
```
34+
35+
Before running Terraform, ensure the GitHub app is configured. See the [configuration details](../../README.md#usages) for more details.
36+
37+
```bash
38+
terraform init
39+
terraform apply
40+
```
41+
42+
You can receive the webhook details by running:
43+
44+
```bash
45+
terraform output -raw webhook_secret
46+
```
47+
48+
Be-aware some shells will print some end of line character `%`.

Diff for: examples/multi-runner/lambdas-download/main.tf

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
locals {
2+
version = "<REPLACE_BY_GITHUB_RELEASE_VERSION>"
3+
}
4+
5+
module "lambdas" {
6+
source = "../../../modules/download-lambda"
7+
lambdas = [
8+
{
9+
name = "webhook"
10+
tag = local.version
11+
},
12+
{
13+
name = "runners"
14+
tag = local.version
15+
},
16+
{
17+
name = "runner-binaries-syncer"
18+
tag = local.version
19+
}
20+
]
21+
}
22+
23+
output "files" {
24+
value = module.lambdas.files
25+
}

0 commit comments

Comments
 (0)