|
| 1 | +# Integration Tests |
| 2 | + |
| 3 | +This document describes the purpose of integration tests as well as acting as a guide |
| 4 | +on what type of changes require integrations tests and how you should write integration tests. |
| 5 | + |
| 6 | +- [What are CDK Integration Tests](#what-are-cdk-integration-tests) |
| 7 | +- [When are integration tests required](#when-are-integration-tests-required) |
| 8 | +- [How to write Integration Tests](#how-to-write-integration-tests) |
| 9 | + - [Creating a test](#creating-a-test) |
| 10 | + - [New L2 Constructs](#new-l2-constructs) |
| 11 | + - [Existing L2 Constructs](#existing-l2-constructs) |
| 12 | + - [Assertions](#assertions) |
| 13 | + |
| 14 | +## What are CDK Integration Tests |
| 15 | + |
| 16 | +All Construct libraries in the CDK code base have integration tests that serve to - |
| 17 | + |
| 18 | +1. Acts as a regression detector. It does this by running `cdk synth` on the integration test and comparing it against |
| 19 | + the `*.expected.json` file. This highlights how a change affects the synthesized stacks. |
| 20 | +2. Allows for a way to verify if the stacks are still valid CloudFormation templates, as part of an intrusive change. |
| 21 | + This is done by running `yarn integ` which will run `cdk deploy` across all of the integration tests in that package. |
| 22 | + If you are developing a new integration test or for some other reason want to work on a single integration test |
| 23 | + over and over again without running through all the integration tests you can do so using |
| 24 | + `yarn integ integ.test-name.js` .Remember to set up AWS credentials before doing this. |
| 25 | +3. (Optionally) Acts as a way to validate that constructs set up the CloudFormation resources as expected. |
| 26 | + A successful CloudFormation deployment does not mean that the resources are set up correctly. |
| 27 | + |
| 28 | + |
| 29 | +## When are Integration Tests Required |
| 30 | + |
| 31 | +The following list contains common scenarios where we _know_ that integration tests are required. |
| 32 | +This is not an exhaustive list and we will, by default, require integration tests for all |
| 33 | +new features unless there is a good reason why one is not needed. |
| 34 | + |
| 35 | +**1. Adding a new feature that is using previously unused CloudFormation resource types** |
| 36 | +For example, adding a new L2 construct for an L1 resource. There should be a new integration test |
| 37 | +to test that the new L2 successfully creates the resources in AWS. |
| 38 | + |
| 39 | +**2. Adding a new feature that is using previously unused (or untested) CloudFormation properties** |
| 40 | +For example, there is an existing L2 construct for a CloudFormation resource and you are adding |
| 41 | +support for a new property. This could be either a new property that has been added to CloudFormation |
| 42 | +or an existing property that the CDK did not have coverage for. You should either update and existing |
| 43 | +integration test to cover this new property or create a new test. |
| 44 | + |
| 45 | +Sometimes the CloudFormation documentation is incorrect or unclear on the correct way to configure |
| 46 | +a property. This can lead to introducing new features that don't actually work. Creating |
| 47 | +an integration test for the new feature can ensure that it works and avoid unnecessary bugs. |
| 48 | + |
| 49 | +**3. Involves configuring resource types across services (i.e. integrations)** |
| 50 | +For example, you are adding functionality that allows for service x to integrate with service y. |
| 51 | +A good example of this is the [aws-stepfunctions-tasks](./packages/@aws-cdk/aws-stepfunctions-tasks) or |
| 52 | +[aws-apigatewayv2-integrations](./packages/@aws-cdk/aws-apigatewayv2-integrations) modules. Both of these |
| 53 | +have L2 constructs that provide functionality to integrate services. |
| 54 | + |
| 55 | +Sometimes these integrations involve configuring/formatting json/vtl or some other type of data. |
| 56 | +For these types of features it is important to create an integration test that not only validates |
| 57 | +that the infrastructure deploys successfully, but that the intended functionality works. This could |
| 58 | +mean deploying the integration test and then manually making an HTTP request or invoking a Lambda function. |
| 59 | + |
| 60 | +**4. Adding a new supported version (e.g. a new [AuroraMysqlEngineVersion](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_rds.AuroraMysqlEngineVersion.html))** |
| 61 | +Sometimes new versions introduce new CloudFormation properties or new required configuration. |
| 62 | +For example Aurora MySQL version 8 introduced a new parameter and was not compatible with the |
| 63 | +existing parameter (see [#19145](https://github.com/aws/aws-cdk/pull/19145)). |
| 64 | + |
| 65 | +**5. Adding any functionality via a [Custom Resource](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources-readme.html)** |
| 66 | +Custom resources involve non-standard functionality and are at a higher risk of introducing bugs. |
| 67 | + |
| 68 | +## How to write Integration Tests |
| 69 | + |
| 70 | +This section will detail how to write integration tests, how they are executed and how to ensure |
| 71 | +you have good test coverage. |
| 72 | + |
| 73 | +### Creating a Test |
| 74 | + |
| 75 | +An integration tests is any file located in the `test/` directory that has a name that starts with `integ.` |
| 76 | +(e.g. `integ.*.ts`). |
| 77 | + |
| 78 | +To create a new integration test, first create a new file, for example `integ.my-new-construct.ts`. |
| 79 | +The contents of this file should be a CDK app. For example, a very simple integration test for a |
| 80 | +Lambda Function would look like this: |
| 81 | + |
| 82 | +_integ.lambda.ts_ |
| 83 | +```ts |
| 84 | +import * as iam from '@aws-cdk/aws-iam'; |
| 85 | +import * as cdk from '@aws-cdk/core'; |
| 86 | +import * as lambda from '../lib'; |
| 87 | + |
| 88 | +const app = new cdk.App(); |
| 89 | + |
| 90 | +const stack = new cdk.Stack(app, 'aws-cdk-lambda-1'); |
| 91 | + |
| 92 | +const fn = new lambda.Function(stack, 'MyLambda', { |
| 93 | + code: new lambda.InlineCode('foo'), |
| 94 | + handler: 'index.handler', |
| 95 | + runtime: lambda.Runtime.NODEJS_10_X, |
| 96 | +}); |
| 97 | + |
| 98 | +app.synth(); |
| 99 | +``` |
| 100 | + |
| 101 | +To run the test you would run: |
| 102 | + |
| 103 | +*Note - filename must be `*.js`* |
| 104 | +``` |
| 105 | +npm run cdk-integ integ.lambda.js |
| 106 | +``` |
| 107 | + |
| 108 | +This will: |
| 109 | +1. Synthesize the CDK app |
| 110 | +2. `cdk deploy` to your AWS account |
| 111 | +3. `cdk destroy` to delete the stack |
| 112 | +4. Save a snapshot of the synthed CloudFormation template to `integ.lambda.expected.json` |
| 113 | + |
| 114 | +Now when you run `npm test` it will synth the integ app and compare the result with the snapshot. |
| 115 | +If the snapshot has changed the same process must be followed to update the snapshot. |
| 116 | + |
| 117 | +### New L2 Constructs |
| 118 | + |
| 119 | +When creating a new L2 construct (or new construct library) it is important to ensure you have a good |
| 120 | +coverage base from which future contributions can build on. |
| 121 | + |
| 122 | +Some general rules to follow are: |
| 123 | + |
| 124 | +- **1 test with all default values** |
| 125 | +One test for each L2 that only populates the required properties. For a Lambda Function this would look like: |
| 126 | + |
| 127 | +```ts |
| 128 | +new lambda.Function(this, 'Handler', { |
| 129 | + code, |
| 130 | + handler, |
| 131 | + runtime, |
| 132 | +}); |
| 133 | +``` |
| 134 | + |
| 135 | +- **1 test with all values provided** |
| 136 | +One test for each L2 that populates non-default properties. Some of this will come down to judgement, but this should |
| 137 | +be based on major functionality. For example, when testing a Lambda Function there are 37 (*at the time of this writing) different |
| 138 | +input parameters. Some of these can be tested together and don't represent large pieces of functionality, |
| 139 | +while others do. |
| 140 | + |
| 141 | +For example, the test for a Lambda Function might look like this. For most of these properties we are probably fine |
| 142 | +testing them together and just testing one of their values. For example we don't gain much by testing a bunch of |
| 143 | +different `memorySize` settings, as long as we test that we can `set` the memorySize then we should be good. |
| 144 | + |
| 145 | +```ts |
| 146 | +new lambda.Function(this, 'Handler', { |
| 147 | + code, |
| 148 | + handler, |
| 149 | + runtime, |
| 150 | + architecture, |
| 151 | + description, |
| 152 | + environment, |
| 153 | + environmentEncryption, |
| 154 | + functionName, |
| 155 | + initialPolicy, |
| 156 | + insightsVersion, |
| 157 | + layers, |
| 158 | + maxEventAge, |
| 159 | + memorySize, |
| 160 | + reservedConcurrentExecutions, |
| 161 | + retryAttempts, |
| 162 | + role, |
| 163 | + timeout, |
| 164 | + tracing, |
| 165 | +}); |
| 166 | +``` |
| 167 | + |
| 168 | +Other parameters might represent larger pieces of functionality and might create other resources for us or configure |
| 169 | +integrations with other services. For these it might make sense to split them out into separate tests so it is easier |
| 170 | +to reason about them. |
| 171 | + |
| 172 | +A couple of examples would be |
| 173 | +(you could also mix in different configurations of the above parameters with each of these): |
| 174 | + |
| 175 | +_testing filesystems_ |
| 176 | +```ts |
| 177 | +new lambda.Function(this, 'Handler', { |
| 178 | + filesystem, |
| 179 | +}); |
| 180 | +``` |
| 181 | + |
| 182 | +_testing event sources_ |
| 183 | +```ts |
| 184 | +new lambda.Function(this, 'Handler', { |
| 185 | + events, |
| 186 | +}); |
| 187 | +``` |
| 188 | + |
| 189 | +_testing VPCs_ |
| 190 | +```ts |
| 191 | +new lambda.Function(this, 'Handler', { |
| 192 | + securityGroups, |
| 193 | + vpc, |
| 194 | + vpcSubnets, |
| 195 | +}); |
| 196 | +``` |
| 197 | + |
| 198 | +### Existing L2 Constructs |
| 199 | + |
| 200 | +Updating an existing L2 Construct could consist of: |
| 201 | + |
| 202 | +1. **Adding coverage for a new (or previously uncovered) CloudFormation property.** |
| 203 | +In this case you would want to either add this new property to an existing integration test or create a new |
| 204 | +integration test. A new integration test is preferred for larger update (e.g. adding VPC connectivity, etc). |
| 205 | + |
| 206 | +2. **Updating functionality for an existing property.** |
| 207 | +In this case you should first check if you are already covered by an existing integration test. If not, then you would follow the |
| 208 | +same process as adding new coverage. |
| 209 | + |
| 210 | +3. **Changing functionality that affects asset bundling** |
| 211 | +Some constructs deal with asset bundling (i.e. `aws-lambda-nodejs`, `aws-lambda-python`, etc). There are some updates that may not |
| 212 | +touch any CloudFormation property, but instead change the way that code is bundled. While these types of changes may not require |
| 213 | +a change to an integration test, you need to make sure that the integration tests and assertions are rerun. |
| 214 | + |
| 215 | +An example of this would be making a change to the way `aws-lambda-nodejs` bundles Lambda code. A couple of things could go wrong that would |
| 216 | +only be caught by rerunning the integration tests. |
| 217 | + |
| 218 | +1. The bundling commands are only running when performing a real synth (not part of unit tests). Running the integration test confirms |
| 219 | +that the actual bundling was not broken. |
| 220 | +2. When deploying Lambda Functions, CloudFormation will only update the Function configuration with the new code, |
| 221 | +but it will not validate that the Lambda function can be invoked. Because of this, it is important to rerun the integration test |
| 222 | +to deploy the Lambda Function _and_ then rerun the assertions to ensure that the function can still be invoked. |
| 223 | + |
| 224 | +### Assertions |
| 225 | +...Coming soon... |
0 commit comments