Skip to content

Commit 046d0fc

Browse files
corymhallcorymhall
authored
chore: upgrade lerna and use nx (#24896)
Updating lerna to the latest version which now uses `nx` under the hood. Apart from just updating the version, this also allows us to take advantage of nx caching. For now this has only been enabled locally. I'm not 100% sure about the `nx` configuration since our repo is a little weird, but I've been running locally with this configuration for probably 6 months and it has worked fine for me. Side note - the repo restructure has caused some weirdness with our tests. I had to make some updates to existing tests, and I do not know why. Closes #<issue number here>. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent a344507 commit 046d0fc

20 files changed

+2084
-1772
lines changed

‎CONTRIBUTING.md

+12-6
Original file line numberDiff line numberDiff line change
@@ -99,12 +99,12 @@ Most contributions only require working on a single package, usually `aws-cdk-li
9999
time, you can execute the following to build it and it's dependencies.
100100

101101
```console
102-
$ cd packages/aws-cdk-lib
103-
$ ../../scripts/buildup
102+
$ npx lerna run build --scope=aws-cdk-lib
104103
```
105104

106-
Note: The `buildup` command is resumable. If your build fails, you can fix the issue and run `buildup --resume` to
107-
resume.
105+
Note: `lerna` uses a local cache by default. If your build fails, you can fix
106+
the issue and run the command again and it will not rerun any previously
107+
successful steps.
108108

109109
At this point, you can run build and test the `aws-cdk-lib` module by running
110110

@@ -121,13 +121,19 @@ However, if you wish to build the entire repository, the following command will
121121

122122
```console
123123
cd <root of the CDK repo>
124-
scripts/foreach.sh yarn build
124+
npx lerna run build
125125
```
126-
Note: The `foreach` command is resumable by default; you must supply `-r` or `--reset` to start a new session.
127126

128127
You are now ready to start contributing to the CDK. See the [Pull Requests](#pull-requests) section on how to make your
129128
changes and submit it as a pull request.
130129

130+
If you want to run a build without using the local cache, provide the
131+
`--skip-nx-cache` flag.
132+
133+
```console
134+
$ npx lerna run build --skip-nx-cache
135+
```
136+
131137
### Pack
132138

133139
As called out in the above sections, the AWS CDK uses jsii to produce polyglot targets. This means that each CDK module

‎build.sh

+24-7
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ runtarget="build"
66
run_tests="true"
77
check_prereqs="true"
88
check_compat="true"
9+
ci="false"
10+
scope=""
911
while [[ "${1:-}" != "" ]]; do
1012
case $1 in
1113
-h|--help)
@@ -27,6 +29,9 @@ while [[ "${1:-}" != "" ]]; do
2729
--skip-compat)
2830
check_compat="false"
2931
;;
32+
--ci)
33+
ci=true
34+
;;
3035
*)
3136
echo "Unrecognized parameter: $1"
3237
exit 1
@@ -48,6 +53,18 @@ fi
4853

4954
echo "============================================================================================="
5055
echo "installing..."
56+
version=$(node -p "require('./package.json').version")
57+
# this is super weird. If you run 'npm install' twice
58+
# and it actually performs an install, then
59+
# node-bundle test will fail with "npm ERR! maxAge must be a number".
60+
# This won't happen in most instances because if nothing changes then npm install
61+
# won't perform an install.
62+
# In the pipeline however, npm install is run once when all the versions are '0.0.0' (via ./scripts/bump-candidate.sh)
63+
# and then `align-versions` is run which updates all the versions to
64+
# (for example) `2.74.0-rc.0` and then npm install is run again here.
65+
if [ "$version" != "0.0.0" ]; then
66+
rm -rf node_modules
67+
fi
5168
yarn install --frozen-lockfile --network-timeout 1000000
5269

5370
fail() {
@@ -72,21 +89,21 @@ node ./scripts/check-yarn-lock.js
7289
BUILD_INDICATOR=".BUILD_COMPLETED"
7390
rm -rf $BUILD_INDICATOR
7491

75-
# Speed up build by reusing calculated tree hashes
76-
# On dev machine, this speeds up the TypeScript part of the build by ~30%.
77-
export MERKLE_BUILD_CACHE=$(mktemp -d)
78-
trap "rm -rf $MERKLE_BUILD_CACHE" EXIT
79-
8092
if [ "$run_tests" == "true" ]; then
81-
runtarget="$runtarget+test"
93+
runtarget="$runtarget,test"
8294
fi
8395

8496
# Limit top-level concurrency to available CPUs - 1 to limit CPU load.
8597
concurrency=$(node -p 'Math.max(1, require("os").cpus().length - 1)')
8698

99+
flags=""
100+
if [ "$ci" == "true" ]; then
101+
flags="--stream --no-progress --skip-nx-cache"
102+
fi
103+
87104
echo "============================================================================================="
88105
echo "building..."
89-
time npx lerna run $bail --stream --concurrency=$concurrency $runtarget || fail
106+
time npx lerna run $bail --concurrency=$concurrency $runtarget $flags || fail
90107

91108
if [ "$check_compat" == "true" ]; then
92109
/bin/bash scripts/check-api-compatibility.sh

‎buildspec-pr.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ phases:
2424
- /bin/bash ./scripts/cache-load.sh
2525
build:
2626
commands:
27-
- /bin/bash ./build.sh
27+
- /bin/bash ./build.sh --ci
2828
# After compilation, run Rosetta (using the cache if available).
2929
# This will print errors, and fail the build if there are compilation errors in any packages marked as 'strict'.
3030
- /bin/bash ./scripts/run-rosetta.sh

‎buildspec.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ phases:
3030
- codebuild-breakpoint
3131
- 'if ${BUMP_CANDIDATE:-false}; then env NODE_OPTIONS="--max-old-space-size=8196 ${NODE_OPTIONS:-}" /bin/bash ./scripts/bump-candidate.sh; fi'
3232
- /bin/bash ./scripts/align-version.sh
33-
- /bin/bash ./build.sh
33+
- /bin/bash ./build.sh --ci
3434
post_build:
3535
commands:
3636
# Short-circuit: Don't run pack if the above build failed.

‎lerna.json

+18-7
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,29 @@
11
{
2-
"lerna": "^4.0.0",
32
"npmClient": "yarn",
43
"useWorkspaces": true,
54
"packages": [
6-
"packages/*",
5+
"packages/aws-cdk-lib",
6+
"packages/cdk-cli-wrapper",
7+
"packages/cdk-assets",
8+
"packages/aws-cdk",
9+
"packages/cdk",
710
"packages/@aws-cdk/*",
11+
"packages/awslint",
812
"packages/@aws-cdk-containers/*",
913
"packages/@aws-cdk-testing/*",
1014
"packages/@aws-cdk/*/lambda-packages/*",
11-
"tools/*",
12-
"tools/@aws-cdk/*",
13-
"scripts/@aws-cdk/script-tests",
14-
"packages/individual-packages/*"
15+
"tools/@aws-cdk/cdk-build-tools",
16+
"tools/@aws-cdk/cdk-release",
17+
"tools/@aws-cdk/cfn2ts",
18+
"tools/@aws-cdk/eslint-plugin",
19+
"tools/@aws-cdk/node-bundle",
20+
"tools/@aws-cdk/pkglint",
21+
"tools/@aws-cdk/pkgtools",
22+
"tools/@aws-cdk/prlint",
23+
"tools/@aws-cdk/yarn-cling",
24+
"scripts/@aws-cdk/script-tests"
1525
],
1626
"rejectCycles": "true",
17-
"version": "0.0.0"
27+
"version": "0.0.0",
28+
"$schema": "node_modules/lerna/schemas/lerna-schema.json"
1829
}

‎nx.json

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
{
2+
"extends": "@nrwl/workspace/presets/npm.json",
3+
"workspaceLayout": { },
4+
"tasksRunnerOptions": {
5+
"default": {
6+
"runner": "@nrwl/workspace/tasks-runners/default",
7+
"options": {
8+
"cacheableOperations": [
9+
"build",
10+
"test",
11+
"lint",
12+
"package",
13+
"prepare"
14+
]
15+
}
16+
}
17+
},
18+
"targetDefaults": {
19+
"build": {
20+
"implicitDependencies": ["aws-cdk-lib"],
21+
"dependsOn": ["^build"],
22+
"inputs": [
23+
"{projectRoot}/lib/!(*.d|*.generated).ts",
24+
"{projectRoot}/test/!(*.d).ts",
25+
"!{workspaceRoot}/**/tsconfig.json",
26+
"!{workspaceRoot}/**/tsconfig.json",
27+
"!{workspaceRoot}/tsconfig.base.json",
28+
"!{workspaceRoot}/**/tsconfig.tsbuildinfo"
29+
],
30+
"outputs": [
31+
"!{projectRoot}/**/*.integ.snapshot/*",
32+
"{projectRoot}/tsconfig.json",
33+
"{projectRoot}/lib/aws-custom-resource/sdk-api-metadata.json",
34+
"{projectRoot}/build-info.json",
35+
"{projectRoot}/lib/**/*.js",
36+
"{projectRoot}/lib/layer.zip",
37+
"{projectRoot}/bin/**/*.js",
38+
"{projectRoot}/lib/**/*.js.map",
39+
"{projectRoot}/lib/**/*.generated.ts",
40+
"{projectRoot}/lib/**/*.d.ts",
41+
"{projectRoot}/test/**/*.d.ts",
42+
"{projectRoot}/test/**/*.js",
43+
"{projectRoot}/.jsii",
44+
"{projectRoot}/spec/*.json",
45+
"{projectRoot}/.warnings.jsii.js"
46+
]
47+
},
48+
"test": {
49+
"dependsOn": ["build"]
50+
},
51+
},
52+
"affected": {
53+
"defaultBase": "main"
54+
},
55+
"pluginsConfig": {
56+
"@nrwl/js": {
57+
"analyzeSourceFiles": false
58+
}
59+
}
60+
}

‎package.json

+21-8
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,7 @@
1212
"pack": "./pack.sh",
1313
"compat": "./scripts/check-api-compatibility.sh",
1414
"bump": "./bump.sh",
15-
"build-all": "tsc -b",
16-
"postinstall": "patch-package --error-on-fail"
15+
"build-all": "tsc -b"
1716
},
1817
"devDependencies": {
1918
"@types/node": "18.11.19",
@@ -28,10 +27,13 @@
2827
"jsii-pacmak": "1.78.1",
2928
"jsii-reflect": "1.78.1",
3029
"jsii-rosetta": "~5.0.0",
31-
"lerna": "^4.0.0",
30+
"lerna": "^6.6.1",
3231
"patch-package": "^6.5.1",
3332
"semver": "^6.3.0",
3433
"standard-version": "^9.5.0",
34+
"@nrwl/cli": "^15.9.1",
35+
"@nrwl/workspace": "^15.9.1",
36+
"nx": "^15.9.1",
3537
"typescript": "~4.9.5"
3638
},
3739
"resolutions": {
@@ -66,15 +68,26 @@
6668
},
6769
"workspaces": {
6870
"packages": [
69-
"packages/*",
71+
"packages/aws-cdk-lib",
72+
"packages/cdk-cli-wrapper",
73+
"packages/aws-cdk",
74+
"packages/cdk",
75+
"packages/cdk-assets",
7076
"packages/@aws-cdk/*",
77+
"packages/awslint",
7178
"packages/@aws-cdk-containers/*",
7279
"packages/@aws-cdk-testing/*",
7380
"packages/@aws-cdk/*/lambda-packages/*",
74-
"tools/*",
75-
"tools/@aws-cdk/*",
76-
"scripts/@aws-cdk/script-tests",
77-
"packages/individual-packages/*"
81+
"tools/@aws-cdk/cdk-build-tools",
82+
"tools/@aws-cdk/cdk-release",
83+
"tools/@aws-cdk/cfn2ts",
84+
"tools/@aws-cdk/eslint-plugin",
85+
"tools/@aws-cdk/node-bundle",
86+
"tools/@aws-cdk/pkglint",
87+
"tools/@aws-cdk/pkgtools",
88+
"tools/@aws-cdk/prlint",
89+
"tools/@aws-cdk/yarn-cling",
90+
"scripts/@aws-cdk/script-tests"
7891
],
7992
"nohoist": [
8093
"**/jszip",

‎packages/@aws-cdk-testing/framework-integ/test/aws-ec2/test/integ.nat-instances.lit.js.snapshot/aws-cdk-vpc-nat-instances.assets.json

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
{
2-
"version": "21.0.0",
2+
"version": "31.0.0",
33
"files": {
4-
"37e0c4dfe9c2afeff7fb5beab5ab205de8e42b55f7c46884d976fe647d43c06b": {
4+
"d28780ff347cb790782f733de5ab866ea07bb7ee12ee3e8876c976146f2073d4": {
55
"source": {
66
"path": "aws-cdk-vpc-nat-instances.template.json",
77
"packaging": "file"
88
},
99
"destinations": {
1010
"12345678-test-region": {
1111
"bucketName": "cdk-hnb659fds-assets-12345678-test-region",
12-
"objectKey": "37e0c4dfe9c2afeff7fb5beab5ab205de8e42b55f7c46884d976fe647d43c06b.json",
12+
"objectKey": "d28780ff347cb790782f733de5ab866ea07bb7ee12ee3e8876c976146f2073d4.json",
1313
"region": "test-region",
1414
"assumeRoleArn": "arn:${AWS::Partition}:iam::12345678:role/cdk-hnb659fds-file-publishing-role-12345678-test-region"
1515
}

‎packages/@aws-cdk-testing/framework-integ/test/aws-ec2/test/integ.nat-instances.lit.js.snapshot/aws-cdk-vpc-nat-instances.template.json

+8-8
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
"VpcId": {
2222
"Ref": "MyVpcF9F0CA6F"
2323
},
24-
"AvailabilityZone": "test-region-1a",
24+
"AvailabilityZone": "dummy1a",
2525
"CidrBlock": "10.0.0.0/19",
2626
"MapPublicIpOnLaunch": true,
2727
"Tags": [
@@ -93,7 +93,7 @@
9393
"MyVpcPublicSubnet1NatInstance8E94E5F7": {
9494
"Type": "AWS::EC2::Instance",
9595
"Properties": {
96-
"AvailabilityZone": "test-region-1a",
96+
"AvailabilityZone": "dummy1a",
9797
"IamInstanceProfile": {
9898
"Ref": "MyVpcPublicSubnet1NatInstanceInstanceProfile2FD934CB"
9999
},
@@ -131,7 +131,7 @@
131131
"VpcId": {
132132
"Ref": "MyVpcF9F0CA6F"
133133
},
134-
"AvailabilityZone": "test-region-1b",
134+
"AvailabilityZone": "dummy1b",
135135
"CidrBlock": "10.0.32.0/19",
136136
"MapPublicIpOnLaunch": true,
137137
"Tags": [
@@ -203,7 +203,7 @@
203203
"MyVpcPublicSubnet2NatInstance04BCE4E3": {
204204
"Type": "AWS::EC2::Instance",
205205
"Properties": {
206-
"AvailabilityZone": "test-region-1b",
206+
"AvailabilityZone": "dummy1b",
207207
"IamInstanceProfile": {
208208
"Ref": "MyVpcPublicSubnet2NatInstanceInstanceProfile5AB09EF6"
209209
},
@@ -241,7 +241,7 @@
241241
"VpcId": {
242242
"Ref": "MyVpcF9F0CA6F"
243243
},
244-
"AvailabilityZone": "test-region-1c",
244+
"AvailabilityZone": "dummy1c",
245245
"CidrBlock": "10.0.64.0/19",
246246
"MapPublicIpOnLaunch": true,
247247
"Tags": [
@@ -306,7 +306,7 @@
306306
"VpcId": {
307307
"Ref": "MyVpcF9F0CA6F"
308308
},
309-
"AvailabilityZone": "test-region-1a",
309+
"AvailabilityZone": "dummy1a",
310310
"CidrBlock": "10.0.96.0/19",
311311
"MapPublicIpOnLaunch": false,
312312
"Tags": [
@@ -368,7 +368,7 @@
368368
"VpcId": {
369369
"Ref": "MyVpcF9F0CA6F"
370370
},
371-
"AvailabilityZone": "test-region-1b",
371+
"AvailabilityZone": "dummy1b",
372372
"CidrBlock": "10.0.128.0/19",
373373
"MapPublicIpOnLaunch": false,
374374
"Tags": [
@@ -430,7 +430,7 @@
430430
"VpcId": {
431431
"Ref": "MyVpcF9F0CA6F"
432432
},
433-
"AvailabilityZone": "test-region-1c",
433+
"AvailabilityZone": "dummy1c",
434434
"CidrBlock": "10.0.160.0/19",
435435
"MapPublicIpOnLaunch": false,
436436
"Tags": [

‎packages/@aws-cdk-testing/framework-integ/test/aws-ec2/test/integ.nat-instances.lit.js.snapshot/cdk.out

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{"version":"20.0.0"}
1+
{"version":"31.0.0"}

‎packages/@aws-cdk-testing/framework-integ/test/aws-ec2/test/integ.nat-instances.lit.js.snapshot/integ.json

+5-7
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,12 @@
11
{
2-
"version": "21.0.0",
2+
"version": "31.0.0",
33
"testCases": {
4-
"integ.nat-instances.lit": {
4+
"integ-test/DefaultTest": {
55
"stacks": [
66
"aws-cdk-vpc-nat-instances"
77
],
8-
"diffAssets": false,
9-
"stackUpdateWorkflow": true
8+
"assertionStack": "integ-test/DefaultTest/DeployAssert",
9+
"assertionStackName": "integtestDefaultTestDeployAssert24D5C536"
1010
}
11-
},
12-
"synthContext": {},
13-
"enableLookups": true
11+
}
1412
}

0 commit comments

Comments
 (0)