feat(cli-integ): calculate maxWorkers based on machine spec (#423)

In order to be able to run on various machine types, calculate the max
number of workers based on the total memory and CPU cores available in
the machine. With this calculation we get:

- CLI Repo (64GB & 16 Cores) = `Math.min(16 * 15, (64000 - 2000) / (0.7
* 500)) = Math.min(240, 177) = 177`
- CodeBuild Canary (16GB & 8 Cores) = `Math.min(8 * 15, (16000 - 2000) /
(0.7 * 500)) = Math.min(120, 40) = 40`
- CodeBuild Release & Main (72GB & 32 Cores) = `Math.min(32 * 15, (72000
- 2000) / (0.7 * 500)) = Math.min(480, 200) = 200`

### Memory Data

Here is some memory usage sampling i've
[added](#422) to confirm our
memory consumption stats:

```console

2025-04-29T09:50:51.525Z | [pre-callback] [test] Memory RSS: 152 MB
2025-04-29T09:50:51.200Z | [pre-callback] [test] Memory RSS: 150 MB
2025-04-29T09:50:50.836Z | [pre-callback] [test] Memory RSS: 150 MB
2025-04-29T09:54:08.055Z | [post-callback] [test] Memory RSS: 122 MB
2025-04-29T09:54:08.797Z | [post-callback] [test] Memory RSS: 114 MB
2025-04-29T09:54:12.054Z | [post-callback] [test] Memory RSS: 113 MB

2025-04-29T09:53:06.601Z | [pre-tree] [app.js] Memory RSS: 49 MB
2025-04-29T09:53:10.468Z | [pre-tree] [app.js] Memory RSS: 49 MB
2025-04-29T09:54:04.682Z | [pre-tree] [app.js] Memory RSS: 49 MB
2025-04-29T09:53:07.836Z | [post-tree] [app.js] Memory RSS: 105 MB
2025-04-29T09:54:05.071Z | [post-tree] [app.js] Memory RSS: 106 MB
2025-04-29T09:53:11.895Z | [post-tree] [app.js] Memory RSS: 105 MB
2025-04-29T09:53:08.864Z | [post-synth] [app.js] Memory RSS: 139 MB
2025-04-29T09:53:13.215Z | [post-synth] [app.js] Memory RSS: 139 MB
2025-04-29T09:54:05.367Z | [post-synth] [app.js] Memory RSS: 138 MB

2025-04-29T09:53:09.219Z | [post-synth] [cli] Memory RSS: 200 MB
2025-04-29T09:53:13.359Z | [post-synth] [cli] Memory RSS: 203 MB
2025-04-29T09:54:05.437Z | [post-synth] [cli] Memory RSS: 202 MB
2025-04-29T09:53:10.358Z | [pre-synth] [cli] Memory RSS: 201 MB
2025-04-29T09:54:04.627Z | [pre-synth] [cli] Memory RSS: 183 MB

```  

From here we see that:

- Memory consumed by jest process: ~150MB
- Memory consumed by CDK App subprocess: ~140MB
- Memory consumed by CLI subprocess: ~200MB

## Notes

- These numbers suggest a total of about 500MB per worker...multiple
that by 50 and you get 25GB - way above the 16GB we were running with.
Stands to reason that a simple 500 X 50 multiplication doesn't
accurately describe the total max memory consumption of the suite.
- Its surprising that the memory consumed by jest after the callback
finishes is less than before the callback starts.

---
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license

---------

Co-authored-by: Momo Kornher <[email protected]>

Author: iliapolo

.github/workflows/integ.yml

@@ -1647,7 +1647,6 @@ new CdkCliIntegTestsWorkflow(repo, {
   testEnvironment: TEST_ENVIRONMENT,
   buildRunsOn: POWERFUL_RUNNER,
   testRunsOn: POWERFUL_RUNNER,
-  maxWorkers: '80',
 
   localPackages: [
     cloudAssemblySchema.name,

.projenrc.ts

@@ -1,4 +1,5 @@
 const path = require('path');
+const os = require('os');
 
 const rootDir = path.resolve(__dirname, '..', 'tests', process.env.TEST_SUITE_NAME);
 
@@ -19,9 +20,42 @@ module.exports = {
   // for the lock. Which is almost never what we actually care about. Set it high.
   testTimeout: 2 * 60 * 60_000,
 
-  maxWorkers: 50,
+  maxWorkers: maxWorkers(),
   reporters: [
     "default",
     ["jest-junit", { suiteName: "jest tests", outputDirectory: "coverage" }]
   ]
 };
+
+/**
+ * Based on the machine spec, calcluate the maximum number of jest workers we can start in parallel.
+ */
+function maxWorkers() {
+
+  const totalMachineMemoryMB = os.totalmem() / 1024 / 1024;
+  const totalMachineCores = os.cpus().length;
+
+  // empirically observed. this includes:
+  // - 150 jest test process
+  // - 140 app synthesis subprocess  
+  // - 200 cli subprocess
+  const maxWorkerMemoryMB = 500;
+
+  // we take a factor of the total because not all 3 subprocess
+  // consume their max theoretical memory at the same time.
+  // 0.7 is an eyeballed factor that seems to work well.
+  const averageWorkerMemoryMB = 0.7 * maxWorkerMemoryMB;
+
+  // leave some memory for the OS and other external processes
+  const reservedMemoryMB = 2000;
+
+  // our processes don't take up much CPU so we allow for a large factor.
+  const cpuScaleFactor = 15;
+
+  const byMemory = Math.floor((totalMachineMemoryMB - reservedMemoryMB) / (averageWorkerMemoryMB));
+  const byCpu = cpuScaleFactor * totalMachineCores;
+
+  const maxWorkers = Math.min(byMemory, byCpu);
+  console.log(`[integ.jest.config] calculated maxWorkers: ${maxWorkers}`)
+  return maxWorkers;
+}