Skip to content

Commit a96cf55

Browse files
authored
fix(cli): template created during import should be written to assets folder (#29830)
### Issue # (if applicable) Closes #22928 #22530 I'm reopening as the original was closed due to inactivity. ### Reason for this change When the template is synthesized, if it is larger than 50kb and cannot be inlined in the CFN request, CLI writes it to the filesystem to be uploaded to S3. Unlike regular deployments `import` overrides the template, when it does so the code responsible for writing the large template to the filesystem writes to the project root. This code reproduces the issue: ``` import * as cdk from "aws-cdk-lib"; import { Construct } from "constructs"; import * as sqs from "aws-cdk-lib/aws-sqs"; import * as iam from "aws-cdk-lib/aws-iam"; export class AwsCdkImportBugStack extends cdk.Stack { constructor(scope: Construct, id: string, props?: cdk.StackProps) { super(scope, id, props); for (let i = 0; i < 70; i++) { const q = new sqs.Queue(this, `AwsCdkImportBugQueue${i}`, { visibilityTimeout: cdk.Duration.seconds(300), enforceSSL: true, }); } // new sqs.Queue(this, `AwsCdkImportBugQueue`, { // visibilityTimeout: cdk.Duration.seconds(300), // enforceSSL: true, // removalPolicy: cdk.RemovalPolicy.RETAIN, // }); } } ``` Deploy the stack, uncomment the queue, and try to import. This error happens: ``` $ npx cdk import AwsCdkImportBugStack AwsCdkImportBugStack/AwsCdkImportBugQueue/Resource (AWS::SQS::Queue): enter QueueUrl (empty to skip): https://sqs.eu-central-1.amazonaws.com/<cut>/AwsCdkBugStack-keep186A2ECA-IznVNwytuY1b AwsCdkImportBugStack/AwsCdkImportBugQueue/Policy/Resource (AWS::SQS::QueuePolicy): enter Id (empty to skip): Skipping import of AwsCdkImportBugStack/AwsCdkImportBugQueue/Policy/Resource AwsCdkImportBugStack: importing resources into stack... [0%] start: Publishing 06a2c6415fd2982e3285e9d615e5f9b99d1d795a9064479fbe6b88455ac62f6c:current [100%] fail: ENOENT: no such file or directory, open '/home/nburtsev/projects/aws-cdk-import-bug/cdk.out/AwsCdkImportBugStack.template.json-06a2c6415fd2982e3285e9d615e5f9b99d1d795a9064479fbe6b88455ac62f6c.yaml' ❌ AwsCdkImportBugStack failed: Error: Failed to publish one or more assets. See the error messages above for more information. at publishAssets (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:420:84876) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async deployStack (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:430:391) at async ResourceImporter.importResources (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:433:171694) at async ResourceImporter.importResourcesFromMap (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:433:171357) at async CdkToolkit.import (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:433:205058) at async exec4 (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:488:54378) Failed to publish one or more assets. See the error messages above for more information. ``` The file is created, it's just in the project root not in cdk.out: ``` ~/projects/aws-cdk-import-bug $ ls -la total 284 drwxr-xr-x 7 nburtsev users 4096 Mar 22 10:48 . drwxr-xr-x 8 nburtsev users 4096 Mar 22 10:11 .. -rw-r--r-- 1 nburtsev users 64131 Mar 22 10:48 AwsCdkImportBugStack.template.json-06a2c6415fd2982e3285e9d615e5f9b99d1d795a9064479fbe6b88455ac62f6c.yaml drwxr-xr-x 2 nburtsev users 4096 Mar 22 10:12 bin -rw-r--r-- 1 nburtsev users 3185 Mar 22 10:12 cdk.json drwxr-xr-x 2 nburtsev users 4096 Mar 22 10:48 cdk.out <cut> ``` ### Description of changes I've just added a path.join similar to how it is done in regular template generation. ### Description of how you validated changes I've added a test case but I believe tests are broken globally as per @TheRealAmazonKendra ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent eb39d9e commit a96cf55

File tree

3 files changed

+71
-19
lines changed

3 files changed

+71
-19
lines changed

packages/@aws-cdk-testing/cli-integ/resources/cdk-apps/app/app.js

+28-1
Original file line numberDiff line numberDiff line change
@@ -248,10 +248,37 @@ class MigrateStack extends cdk.Stack {
248248
}
249249
}
250250

251-
class ImportableStack extends MigrateStack {
251+
class ImportableStack extends cdk.Stack {
252252
constructor(parent, id, props) {
253253
super(parent, id, props);
254254
new cdk.CfnWaitConditionHandle(this, 'Handle');
255+
256+
if (process.env.INCLUDE_SINGLE_QUEUE === '1') {
257+
const queue = new sqs.Queue(this, 'Queue', {
258+
removalPolicy: (process.env.RETAIN_SINGLE_QUEUE === '1') ? cdk.RemovalPolicy.RETAIN : cdk.RemovalPolicy.DESTROY,
259+
});
260+
261+
new cdk.CfnOutput(this, 'QueueName', {
262+
value: queue.queueName,
263+
});
264+
265+
new cdk.CfnOutput(this, 'QueueUrl', {
266+
value: queue.queueUrl,
267+
});
268+
269+
new cdk.CfnOutput(this, 'QueueLogicalId', {
270+
value: queue.node.defaultChild.logicalId,
271+
});
272+
}
273+
274+
if (process.env.LARGE_TEMPLATE === '1') {
275+
for (let i = 1; i <= 70; i++) {
276+
new sqs.Queue(this, `cdk-import-queue-test${i}`, {
277+
enforceSSL: true,
278+
removalPolicy: cdk.RemovalPolicy.DESTROY,
279+
});
280+
}
281+
}
255282
}
256283
}
257284

packages/@aws-cdk-testing/cli-integ/tests/cli-integ-tests/cli.integtest.ts

+40-17
Original file line numberDiff line numberDiff line change
@@ -1619,37 +1619,60 @@ integTest('skips notice refresh', withDefaultFixture(async (fixture) => {
16191619
}));
16201620

16211621
/**
1622-
* Create a queue with a fresh name, redeploy orphaning the queue, then import it again
1622+
* Create a queue, orphan that queue, then import the queue.
1623+
*
1624+
* We want to test with a large template to make sure large templates can work with import.
16231625
*/
16241626
integTest('test resource import', withDefaultFixture(async (fixture) => {
1625-
const outputsFile = path.join(fixture.integTestDir, 'outputs', 'outputs.json');
1627+
// GIVEN
1628+
const randomPrefix = randomString();
1629+
const uniqueOutputsFileName = `${randomPrefix}Outputs.json`; // other tests use the outputs file. Make sure we don't collide.
1630+
const outputsFile = path.join(fixture.integTestDir, 'outputs', uniqueOutputsFileName);
16261631
await fs.mkdir(path.dirname(outputsFile), { recursive: true });
16271632

1628-
// Initial deploy
1633+
// First, create a stack that includes many queues, and one queue that will be removed from the stack but NOT deleted from AWS.
16291634
await fixture.cdkDeploy('importable-stack', {
1630-
modEnv: { ORPHAN_TOPIC: '1' },
1635+
modEnv: { LARGE_TEMPLATE: '1', INCLUDE_SINGLE_QUEUE: '1', RETAIN_SINGLE_QUEUE: '1' },
16311636
options: ['--outputs-file', outputsFile],
16321637
});
16331638

1634-
const outputs = JSON.parse((await fs.readFile(outputsFile, { encoding: 'utf-8' })).toString());
1635-
const queueName = outputs.QueueName;
1636-
const queueLogicalId = outputs.QueueLogicalId;
1637-
fixture.log(`Setup complete, created queue ${queueName}`);
16381639
try {
1639-
// Deploy again, orphaning the queue
1640+
1641+
// Second, now the queue we will remove is in the stack and has a logicalId. We can now make the resource mapping file.
1642+
// This resource mapping file will be used to tell the import operation what queue to bring into the stack.
1643+
const fullStackName = fixture.fullStackName('importable-stack');
1644+
const outputs = JSON.parse((await fs.readFile(outputsFile, { encoding: 'utf-8' })).toString());
1645+
const queueLogicalId = outputs[fullStackName].QueueLogicalId;
1646+
const queueResourceMap = {
1647+
[queueLogicalId]: { QueueUrl: outputs[fullStackName].QueueUrl },
1648+
};
1649+
const mappingFile = path.join(fixture.integTestDir, 'outputs', `${randomPrefix}Mapping.json`);
1650+
await fs.writeFile(
1651+
mappingFile,
1652+
JSON.stringify(queueResourceMap),
1653+
{ encoding: 'utf-8' },
1654+
);
1655+
1656+
// Third, remove the queue from the stack, but don't delete the queue from AWS.
16401657
await fixture.cdkDeploy('importable-stack', {
1641-
modEnv: { OMIT_TOPIC: '1' },
1658+
modEnv: { LARGE_TEMPLATE: '1', INCLUDE_SINGLE_QUEUE: '0', RETAIN_SINGLE_QUEUE: '0' },
16421659
});
1660+
const cfnTemplateBeforeImport = await fixture.aws.cloudFormation('getTemplate', { StackName: fullStackName });
1661+
expect(cfnTemplateBeforeImport.TemplateBody).not.toContain(queueLogicalId);
16431662

1644-
// Write a resource mapping file based on the ID from step one, then run an import
1645-
const mappingFile = path.join(fixture.integTestDir, 'outputs', 'mapping.json');
1646-
await fs.writeFile(mappingFile, JSON.stringify({ [queueLogicalId]: { QueueName: queueName } }), { encoding: 'utf-8' });
1663+
// WHEN
1664+
await fixture.cdk(
1665+
['import', '--resource-mapping', mappingFile, fixture.fullStackName('importable-stack')],
1666+
{ modEnv: { LARGE_TEMPLATE: '1', INCLUDE_SINGLE_QUEUE: '1', RETAIN_SINGLE_QUEUE: '0' } },
1667+
);
16471668

1648-
await fixture.cdk(['import',
1649-
'--resource-mapping', mappingFile,
1650-
fixture.fullStackName('importable-stack')]);
1669+
// THEN
1670+
const describeStacksResponse = await fixture.aws.cloudFormation('describeStacks', { StackName: fullStackName });
1671+
const cfnTemplateAfterImport = await fixture.aws.cloudFormation('getTemplate', { StackName: fullStackName });
1672+
expect(describeStacksResponse.Stacks![0].StackStatus).toEqual('IMPORT_COMPLETE');
1673+
expect(cfnTemplateAfterImport.TemplateBody).toContain(queueLogicalId);
16511674
} finally {
1652-
// Cleanup
1675+
// Clean up
16531676
await fixture.cdkDestroy('importable-stack');
16541677
}
16551678
}));

packages/aws-cdk/lib/api/util/template-body-parameter.ts

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import * as path from 'path';
12
import * as cxapi from '@aws-cdk/cx-api';
23
import * as chalk from 'chalk';
34
import * as fs from 'fs-extra';
@@ -68,7 +69,8 @@ export async function makeBodyParameter(
6869
if (overrideTemplate) {
6970
// Add a variant of this template
7071
templateFile = `${stack.templateFile}-${templateHash}.yaml`;
71-
await fs.writeFile(templateFile, templateJson, { encoding: 'utf-8' });
72+
const templateFilePath = path.join(stack.assembly.directory, templateFile);
73+
await fs.writeFile(templateFilePath, templateJson, { encoding: 'utf-8' });
7274
}
7375

7476
assetManifest.addFileAsset(templateHash, {

0 commit comments

Comments
 (0)