Skip to content

chore: run e2e tests on demand #441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 45 commits into from
Jan 14, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
c3b427d
add AWS infra to run E2E tests
flochaz Jan 6, 2022
27bfa90
Fix e2e tests and add missing scripts
flochaz Jan 6, 2022
cd7b015
split workflow
flochaz Jan 6, 2022
7015f64
Fix region and account setting for tracing e2e
flochaz Jan 7, 2022
d1acd31
add log to tracing e2e
flochaz Jan 7, 2022
b367d80
alternative to get account
flochaz Jan 7, 2022
52a20d8
revert increase timeout
flochaz Jan 7, 2022
c0a4301
remove test trigger
flochaz Jan 7, 2022
38fdb75
remove aws-infra
flochaz Jan 7, 2022
ea45fdf
add Contributing
flochaz Jan 7, 2022
bdf800f
fix on merge
flochaz Jan 7, 2022
e3a83ce
chore: added jest group runner dep + config + headers to logger
dreamorosi Jan 7, 2022
6db56fa
chore: added group filter in regular cmd tests for logger
dreamorosi Jan 7, 2022
6046c08
Add manual command for get trace summary
flochaz Jan 7, 2022
08e8050
Merge remote-tracking branch 'origin/main' into chore/cicd/automateIn…
flochaz Jan 7, 2022
ff9a87f
Update CONTRIBUTING.md
flochaz Jan 7, 2022
13103b8
Update CONTRIBUTING.md
flochaz Jan 7, 2022
3a189ba
Update CONTRIBUTING.md
flochaz Jan 7, 2022
5fbdc66
Update CONTRIBUTING.md
flochaz Jan 7, 2022
f86e54f
Update CONTRIBUTING.md
flochaz Jan 7, 2022
065e538
Update CONTRIBUTING.md
flochaz Jan 7, 2022
29ed2e7
Update CONTRIBUTING.md
flochaz Jan 7, 2022
0cfb133
Update CONTRIBUTING.md
flochaz Jan 7, 2022
92f1183
remove useless steps in owrkflow
flochaz Jan 11, 2022
e196ae1
force test run TO BE REVERTED
flochaz Jan 11, 2022
9c1e339
Revert "force test run TO BE REVERTED"
flochaz Jan 11, 2022
33d87f7
Update CONTRIBUTING.md
flochaz Jan 12, 2022
19c10c4
Update CONTRIBUTING.md
flochaz Jan 12, 2022
f792e5c
Update CONTRIBUTING.md
flochaz Jan 12, 2022
ba91ab7
Update CONTRIBUTING.md
flochaz Jan 12, 2022
51677d8
Update CONTRIBUTING.md
flochaz Jan 12, 2022
9f9b6d6
Update .github/workflows/run-e2e-tests.yml
flochaz Jan 12, 2022
d0f9736
Update CONTRIBUTING.md
flochaz Jan 12, 2022
530d50d
Update .github/workflows/run-e2e-tests.yml
flochaz Jan 12, 2022
df4e51a
Update CONTRIBUTING.md
flochaz Jan 12, 2022
5ff1931
Update CONTRIBUTING.md
flochaz Jan 12, 2022
9d0daa8
Update CONTRIBUTING.md
flochaz Jan 12, 2022
1d22541
Update CONTRIBUTING.md
flochaz Jan 12, 2022
1e4c897
Update CONTRIBUTING.md
flochaz Jan 13, 2022
ee00ffc
Update CONTRIBUTING.md
flochaz Jan 13, 2022
57c3bb0
Update CONTRIBUTING.md
flochaz Jan 13, 2022
0de1d03
Update CONTRIBUTING.md
flochaz Jan 13, 2022
64304ce
homogenize tests run process
flochaz Jan 13, 2022
4f62c1a
fix indentation
flochaz Jan 13, 2022
973a9d5
Update CONTRIBUTING.md
dreamorosi Jan 14, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 31 additions & 16 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@ You can run each group separately or all together thanks to [jest-runner-groups]

Unit tests, under `tests/unit` folder are standard [Jest](https://jestjs.io) tests.

End-to-end tests, under `tests/e2e` folder, will test the module features by deploying AWS Lambda functions into your AWS Account. We use CDK lib for Typescript for creating infrastructure, and `aws sdk` for invoking the functions and assert on the expected behaviors. All steps are also executed by Jest.
End-to-end tests, under `tests/e2e` folder, will test the module features by deploying AWS Lambda functions into your AWS Account. We use [aws-cdk](https://docs.aws.amazon.com/cdk/v1/guide/getting_started.html) v1 library (not v2 due to [this cdk issue](https://github.com/aws/aws-cdk/issues/18211)) for Typescript for creating infrastructure, and [aws-sdk-js v2](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/) for invoking the functions and assert on the expected behaviors. All steps are also executed by Jest.


Running end-to-end tests will deploy AWS resources. You will need an AWS account and the tests might incur costs. The cost from **some services** are covered by the [AWS Free Tier](https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=*all) but not all of them. If you don't have an AWS Account follow [these instructions to create one](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).

Expand All @@ -88,9 +89,10 @@ As mentioned before, tests are split into groups thanks to [jest-runner-groups](

**Run**

`npm run test`

You can run selective tests by restricting the group to the one you want. For instance `npx jest --group=unit/metrics/all`.
To run unit tests you can either use
* npm task `lerna-test:unit` (`npm run lerna-test:unit`) in root folder to run them all
* npm task `test:e2e` (`npm run test:unit`) in module folder (for example: `packages/metrics`) to run the module specific one
* jest directly `npx jest --group=unit` in module folder to run the module specific one (You can run selective tests by restricting the group to the one you want. For instance `npx jest --group=unit/metrics/class`)

**e2e tests**

Expand All @@ -106,30 +108,43 @@ As mentioned in the previous section, tests are split into groups thanks to [jes
*/
```

and leverage [aws-cdk](https://docs.aws.amazon.com/cdk/v1/guide/getting_started.html) V1 package (not v2 due to [this cdk issue](https://github.com/aws/aws-cdk/issues/18211)) to programatically deploy and destroy stacks. See `metrics/tests/e2e/decorator.test.ts` as an example.
See `metrics/tests/e2e/decorator.test.ts` as an example.


**Run**

To run unit tests you can either use projen task
* `npm run test:e2e` which will only run jest integ tests
* or jest directly `npx jest --group=e2e`

You can run selective tests by restricting the group to the one you want. For instance `npx jest --group=e2e/other/example`.
To run e2e tests you can either use
* npm task `lerna-test:e2e` (`npm run lerna-test:e2e`) in root folder
* npm task `test:e2e` (`npm run test:e2e`) in module folder (for example: `packages/metrics`) to run the module specific one
* jest directly `npx jest --group=e2e` in module folder. (You can run selective tests by restricting the group to the one you want. For instance `npx jest --group=e2e/metrics/decorator`)

Two important env variable can be used:
* `AWS_PROFILE` to use the right credentials
* `AWS_PROFILE` to use the right AWS credentials
* `DISABLE_TEARDOWN` if you don't want your stack to be destroyed at the end of the test (useful in dev mode when iterating over your code).

Example: `DISABLE_TEARDOWN=true AWS_PROFILE=ara npx jest --group=integ/other/example`
Example: `DISABLE_TEARDOWN=true AWS_PROFILE=dev-account npx jest --group=e2e/metrics/decorator`

**Automate**

You can run the end-to-end tests automatically on your forked project by following these steps:
1. Create AWS Role
As mention earlier we are leveraging CDK to deploy and clean resources on AWS. Therefore to run those tests through github actions you will need to grant specific permissions to your workflow. To do so you can leverage [@pahud/cdk-github-oidc](https://constructs.dev/packages/@pahud/cdk-github-oidc) construct which setup the right resources to leverage [Github OpenID Connect](https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/) mechanism.
1. Add your new role into your Github fork secrets under `AWS_ROLE_ARN_TO_ASSUME`.
1. Run manually `run-e2e-tests` workflow.
1. Create an IAM role in your target AWS account, with the least amount of privilege.

As mentioned above in this page, we are leveraging CDK to deploy and consequently clean-up resources on AWS. Therefore to run those tests through Github actions you will need to grant specific permissions to your workflow.

We recommend following [Amazon IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) for the AWS credentials used in GitHub Actions workflows, including:
* Do not store credentials in your repository's code.
* [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) to the credentials used in GitHub Actions workflows. Grant only the permissions required to perform the actions in your GitHub Actions workflows.
* [Monitor the activity](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#keep-a-log) of the credentials used in GitHub Actions workflows.

For an example of how to create a role in CDK, you can look at [@pahud/cdk-github-oidc](https://constructs.dev/packages/@pahud/cdk-github-oidc) construct.

More information about:

- [Github OpenID Connect](https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/
- ["Configure AWS Credentials" Action For GitHub Actions](https://github.com/aws-actions/configure-aws-credentials/)
1. Add your new role into your [Github fork secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) with name `AWS_ROLE_ARN_TO_ASSUME`.
1. In your forked repository, go to the "Actions" tabs, select the `run-e2e-tests` workflow.
1. In the run-e2e-tests workflow page, select "Run workflow" and run it on the desired branch.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: steps numbers are all "1."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They rendered correctly by GitHub preview & according to spec it's not required for them to be in order:

To create an ordered list, add line items with numbers followed by periods. The numbers don’t have to be in numerical order, but the list should start with the number one.

But for those who are reading the source I agree that it was confusing, changed it in 973a9d5.


> :Warning: **Don't automatically run end-to-end tests on branch push or PRs**. A malicious attacker can submit a pull request to attack your AWS account. Ideally, use a blank account without any important workload/data, and limit `AWS_ROLE_ARN_TO_ASSUME` permission to least minimum privilege.
### Conventions
Expand Down
3 changes: 2 additions & 1 deletion packages/commons/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
},
"scripts": {
"commit": "commit",
"test": "jest --detectOpenHandles --coverage --verbose",
"test": "npm run test:unit",
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it!

"test:e2e": "echo 'Not Applicable'",
"watch": "jest --watch",
"build": "tsc",
Expand Down
3 changes: 2 additions & 1 deletion packages/logger/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
},
"scripts": {
"commit": "commit",
"test": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test": "npm run test:unit",
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test:e2e": "jest --group=e2e",
"watch": "jest --watch",
"build": "tsc",
Expand Down
3 changes: 2 additions & 1 deletion packages/metrics/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
},
"scripts": {
"commit": "commit",
"test": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test": "npm run test:unit",
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test:e2e": "jest --group=e2e",
"watch": "jest --group=unit --watch ",
"build": "tsc",
Expand Down
3 changes: 2 additions & 1 deletion packages/tracing/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
},
"scripts": {
"commit": "commit",
"test": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test": "npm run test:unit",
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test:e2e": "jest --group=e2e",
"watch": "jest --watch",
"build": "tsc",
Expand Down