feat(cli): preview of cdk import (#17666)

An initial version of `cdk import`, bringing existing resources under the
management of CloudFormation.

To use:

- Make sure your diff is clean (you've recently deployed)
- Add constructs for the resource(s) you want to import. **Make sure the CDK code configures them exactly as they are configured in reality**. 
  - You can provide resource names here but it's probably better if you don't.
- Run `cdk import`
- Provide the actual resource names for each resource (if necessary).
- An importing changeset will execute and the resources are imported.

This is an implementation of aws/aws-cdk-rfcs#52

Author: tomas-mazak

packages/aws-cdk/README.md

@@ -19,6 +19,7 @@ Command                               | Description
 [`cdk synth`](#cdk-synthesize)        | Synthesize a CDK app to CloudFormation template(s)
 [`cdk diff`](#cdk-diff)               | Diff stacks against current state
 [`cdk deploy`](#cdk-deploy)           | Deploy a stack into an AWS account
+[`cdk import`](#cdk-import)           | Import existing AWS resources into a CDK stack
 [`cdk watch`](#cdk-watch)             | Watches a CDK app for deployable and hotswappable changes
 [`cdk destroy`](#cdk-destroy)         | Deletes a stack from an AWS account
 [`cdk bootstrap`](#cdk-bootstrap)     | Deploy a toolkit stack to support deploying large stacks & artifacts
@@ -450,6 +451,53 @@ $ cdk watch --no-logs
 **Note**: This command is considered experimental,
 and might have breaking changes in the future.
 
+### `cdk import`
+
+Sometimes you want to import AWS resources that were created using other means
+into a CDK stack. For some resources (like Roles, Lambda Functions, Event Rules,
+...), it's feasible to create new versions in CDK and then delete the old
+versions. For other resources, this is not possible: stateful resources like S3
+Buckets, DynamoDB tables, etc., cannot be easily deleted without impact on the
+service.
+
+`cdk import`, which uses [CloudFormation resource
+imports](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html),
+makes it possible to bring an existing resource under CDK/CloudFormation's
+management. See the [list of resources that can be imported here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-supported-resources.html).
+
+To import an existing resource to a CDK stack, follow the following steps:
+
+1. Run a `cdk diff` to make sure there are no pending changes to the CDK stack you want to
+   import resources into. The only changes allowed in an "import" operation are
+   the addition of new resources which you want to import.
+2. Add constructs for the resources you want to import to your Stack (for example,
+   for an S3 bucket, add something like `new s3.Bucket(this, 'ImportedS3Bucket', {});`).
+   **Do not add any other changes!** You must also make sure to exactly model the state
+   that the resource currently has. For the example of the Bucket, be sure to
+   include KMS keys, life cycle policies, and anything else that's relevant
+   about the bucket. If you do not, subsequent update operations may not do what
+   you expect.
+3. Run the `cdk import` - if there are multiple stacks in the CDK app, pass a specific
+   stack name as an argument.
+4. The CLI will prompt you to pass in the actual names of the resources you are
+   importing. After you supply it, the import starts.
+5. When `cdk import` reports success, the resource is managed by CDK. Any subsequent
+   changes in the construct configuration will be reflected on the resource.
+
+#### Limitations
+
+This feature is currently in preview. Be aware of the following limitations:
+
+- Importing resources in nested stacks is not possible.
+- Uses the deploy role credentials (necessary to read the encrypted staging
+  bucket). Requires a new version (version 12) of the bootstrap stack, for the added
+  IAM permissions to the `deploy-role`.
+- There is no check on whether the properties you specify are correct and complete
+  for the imported resource. Try starting a drift detection operation after importing.
+- Resources that depend on other resources must all be imported together, or one-by-one
+  in the right order. The CLI will not help you import dependent resources in the right
+  order, the CloudFormation deployment will fail with unresolved references.
+
 ### `cdk destroy`
 
 Deletes a stack from it's environment. This will cause the resources in the stack to be destroyed (unless they were
@@ -521,10 +569,10 @@ NOTICES
 16603   Toggling off auto_delete_objects for Bucket empties the bucket
 
         Overview: If a stack is deployed with an S3 bucket with
-                  auto_delete_objects=True, and then re-deployed with 
-                  auto_delete_objects=False, all the objects in the bucket 
+                  auto_delete_objects=True, and then re-deployed with
+                  auto_delete_objects=False, all the objects in the bucket
                   will be deleted.
-                 
+
         Affected versions: <1.126.0.
 
         More information at: https://github.com/aws/aws-cdk/issues/16603
@@ -533,12 +581,12 @@ NOTICES
 17061   Error when building EKS cluster with monocdk import
 
         Overview: When using monocdk/aws-eks to build a stack containing
-                  an EKS cluster, error is thrown about missing 
+                  an EKS cluster, error is thrown about missing
                   lambda-layer-node-proxy-agent/layer/package.json.
-         
+
         Affected versions: >=1.126.0 <=1.130.0.
 
-        More information at: https://github.com/aws/aws-cdk/issues/17061 
+        More information at: https://github.com/aws/aws-cdk/issues/17061
 
 
 If you don’t want to see an notice anymore, use "cdk acknowledge ID". For example, "cdk acknowledge 16603".
@@ -580,7 +628,7 @@ $cdk acknowledge 16603
 ### `cdk notices`
 
 List the notices that are relevant to the current CDK repository, regardless of context flags or notices that
-have been acknowledged: 
+have been acknowledged:
 
 ```console
 $ cdk notices
@@ -589,9 +637,9 @@ NOTICES
 
 16603   Toggling off auto_delete_objects for Bucket empties the bucket
 
-        Overview: if a stack is deployed with an S3 bucket with 
-                  auto_delete_objects=True, and then re-deployed with 
-                  auto_delete_objects=False, all the objects in the bucket 
+        Overview: if a stack is deployed with an S3 bucket with
+                  auto_delete_objects=True, and then re-deployed with
+                  auto_delete_objects=False, all the objects in the bucket
                   will be deleted.
 
         Affected versions: framework: <=2.15.0 >=2.10.0

packages/aws-cdk/lib/api/bootstrap/bootstrap-template.yaml

@@ -453,6 +453,8 @@ Resources:
                   - cloudformation:DeleteStack
                   - cloudformation:UpdateTerminationProtection
                   - sts:GetCallerIdentity
+                  # `cdk import`
+                  - cloudformation:GetTemplateSummary
                 Resource: "*"
                 Effect: Allow
               - Sid: CliStagingBucket
@@ -507,7 +509,7 @@ Resources:
       Type: String
       Name:
         Fn::Sub: '/cdk-bootstrap/${Qualifier}/version'
-      Value: '11'
+      Value: '12'
 Outputs:
   BucketName:
     Description: The name of the S3 bucket owned by the CDK toolkit stack

packages/aws-cdk/lib/api/cloudformation-deployments.ts

@@ -6,10 +6,10 @@ import { publishAssets } from '../util/asset-publishing';
 import { Mode } from './aws-auth/credentials';
 import { ISDK } from './aws-auth/sdk';
 import { SdkProvider } from './aws-auth/sdk-provider';
-import { deployStack, DeployStackResult, destroyStack } from './deploy-stack';
+import { deployStack, DeployStackResult, destroyStack, makeBodyParameterAndUpload } from './deploy-stack';
 import { loadCurrentTemplateWithNestedStacks, loadCurrentTemplate } from './nested-stack-helpers';
 import { ToolkitInfo } from './toolkit-info';
-import { CloudFormationStack, Template } from './util/cloudformation';
+import { CloudFormationStack, Template, ResourcesToImport, ResourceIdentifierSummaries } from './util/cloudformation';
 import { StackActivityProgress } from './util/cloudformation/stack-activity-monitor';
 import { replaceEnvPlaceholders } from './util/placeholders';
 
@@ -224,6 +224,18 @@ export interface DeployStackOptions {
    * @default - nothing extra is appended to the User-Agent header
    */
   readonly extraUserAgent?: string;
+
+  /**
+   * List of existing resources to be IMPORTED into the stack, instead of being CREATED
+   */
+  readonly resourcesToImport?: ResourcesToImport;
+
+  /**
+   * If present, use this given template instead of the stored one
+   *
+   * @default - Use the stored template
+   */
+  readonly overrideTemplate?: any;
 }
 
 export interface DestroyStackOptions {
@@ -280,23 +292,52 @@ export class CloudFormationDeployments {
   }
 
   public async readCurrentTemplateWithNestedStacks(rootStackArtifact: cxapi.CloudFormationStackArtifact): Promise<Template> {
-    const sdk = await this.prepareSdkWithLookupOrDeployRole(rootStackArtifact);
+    const sdk = (await this.prepareSdkWithLookupOrDeployRole(rootStackArtifact)).stackSdk;
     return (await loadCurrentTemplateWithNestedStacks(rootStackArtifact, sdk)).deployedTemplate;
   }
 
   public async readCurrentTemplate(stackArtifact: cxapi.CloudFormationStackArtifact): Promise<Template> {
     debug(`Reading existing template for stack ${stackArtifact.displayName}.`);
-    const sdk = await this.prepareSdkWithLookupOrDeployRole(stackArtifact);
+    const sdk = (await this.prepareSdkWithLookupOrDeployRole(stackArtifact)).stackSdk;
     return loadCurrentTemplate(stackArtifact, sdk);
   }
 
+  public async resourceIdentifierSummaries(
+    stackArtifact: cxapi.CloudFormationStackArtifact,
+    toolkitStackName?: string,
+  ): Promise<ResourceIdentifierSummaries> {
+    debug(`Retrieving template summary for stack ${stackArtifact.displayName}.`);
+    // Currently, needs to use `deploy-role` since it may need to read templates in the staging
+    // bucket which have been encrypted with a KMS key (and lookup-role may not read encrypted things)
+    const { stackSdk, resolvedEnvironment } = await this.prepareSdkFor(stackArtifact, undefined, Mode.ForReading);
+    const cfn = stackSdk.cloudFormation();
+
+    const toolkitInfo = await ToolkitInfo.lookup(resolvedEnvironment, stackSdk, toolkitStackName);
+
+    // Upload the template, if necessary, before passing it to CFN
+    const cfnParam = await makeBodyParameterAndUpload(
+      stackArtifact,
+      resolvedEnvironment,
+      toolkitInfo,
+      this.sdkProvider,
+      stackSdk);
+
+    const response = await cfn.getTemplateSummary(cfnParam).promise();
+    if (!response.ResourceIdentifierSummaries) {
+      debug('GetTemplateSummary API call did not return "ResourceIdentifierSummaries"');
+    }
+    return response.ResourceIdentifierSummaries ?? [];
+  }
+
   public async deployStack(options: DeployStackOptions): Promise<DeployStackResult> {
     const { stackSdk, resolvedEnvironment, cloudFormationRoleArn } = await this.prepareSdkFor(options.stack, options.roleArn);
 
     const toolkitInfo = await ToolkitInfo.lookup(resolvedEnvironment, stackSdk, options.toolkitStackName);
 
-    // Publish any assets before doing the actual deploy
-    await this.publishStackAssets(options.stack, toolkitInfo);
+    // Publish any assets before doing the actual deploy (do not publish any assets on import operation)
+    if (options.resourcesToImport === undefined) {
+      await this.publishStackAssets(options.stack, toolkitInfo);
+    }
 
     // Do a verification of the bootstrap stack version
     await this.validateBootstrapStackVersion(
@@ -327,6 +368,8 @@ export class CloudFormationDeployments {
       rollback: options.rollback,
       hotswap: options.hotswap,
       extraUserAgent: options.extraUserAgent,
+      resourcesToImport: options.resourcesToImport,
+      overrideTemplate: options.overrideTemplate,
     });
   }
 
@@ -348,16 +391,19 @@ export class CloudFormationDeployments {
     return stack.exists;
   }
 
-  private async prepareSdkWithLookupOrDeployRole(stackArtifact: cxapi.CloudFormationStackArtifact): Promise<ISDK> {
+  private async prepareSdkWithLookupOrDeployRole(stackArtifact: cxapi.CloudFormationStackArtifact): Promise<PreparedSdkForEnvironment> {
     // try to assume the lookup role
     try {
       const result = await prepareSdkWithLookupRoleFor(this.sdkProvider, stackArtifact);
       if (result.didAssumeRole) {
-        return result.sdk;
+        return {
+          resolvedEnvironment: result.resolvedEnvironment,
+          stackSdk: result.sdk,
+        };
       }
     } catch { }
     // fall back to the deploy role
-    return (await this.prepareSdkFor(stackArtifact, undefined, Mode.ForReading)).stackSdk;
+    return this.prepareSdkFor(stackArtifact, undefined, Mode.ForReading);
   }
 
   /**

packages/aws-cdk/lib/api/deploy-stack.ts

@@ -1,5 +1,6 @@
 import * as cxapi from '@aws-cdk/cx-api';
 import * as chalk from 'chalk';
+import * as fs from 'fs-extra';
 import * as uuid from 'uuid';
 import { addMetadataAssetsToManifest } from '../assets';
 import { Tag } from '../cdk-toolkit';
@@ -15,7 +16,7 @@ import { ICON } from './hotswap/common';
 import { ToolkitInfo } from './toolkit-info';
 import {
   changeSetHasNoChanges, CloudFormationStack, TemplateParameters, waitForChangeSet,
-  waitForStackDeploy, waitForStackDelete, ParameterValues, ParameterChanges,
+  waitForStackDeploy, waitForStackDelete, ParameterValues, ParameterChanges, ResourcesToImport,
 } from './util/cloudformation';
 import { StackActivityMonitor, StackActivityProgress } from './util/cloudformation/stack-activity-monitor';
 
@@ -189,6 +190,19 @@ export interface DeployStackOptions {
    * @default - nothing extra is appended to the User-Agent header
    */
   readonly extraUserAgent?: string;
+
+  /**
+   * If set, change set of type IMPORT will be created, and resourcesToImport
+   * passed to it.
+   */
+  readonly resourcesToImport?: ResourcesToImport;
+
+  /**
+   * If present, use this given template instead of the stored one
+   *
+   * @default - Use the stored template
+   */
+  readonly overrideTemplate?: any;
 }
 
 const LARGE_TEMPLATE_SIZE_KB = 50;
@@ -245,7 +259,13 @@ export async function deployStack(options: DeployStackOptions): Promise<DeploySt
     debug(`${deployName}: deploying...`);
   }
 
-  const bodyParameter = await makeBodyParameter(stackArtifact, options.resolvedEnvironment, legacyAssets, options.toolkitInfo, options.sdk);
+  const bodyParameter = await makeBodyParameter(
+    stackArtifact,
+    options.resolvedEnvironment,
+    legacyAssets,
+    options.toolkitInfo,
+    options.sdk,
+    options.overrideTemplate);
   await publishAssets(legacyAssets.toManifest(stackArtifact.assembly.directory), options.sdkProvider, stackEnv);
 
   if (options.hotswap) {
@@ -298,7 +318,8 @@ async function prepareAndExecuteChangeSet(
   const changeSet = await cfn.createChangeSet({
     StackName: deployName,
     ChangeSetName: changeSetName,
-    ChangeSetType: update ? 'UPDATE' : 'CREATE',
+    ChangeSetType: options.resourcesToImport ? 'IMPORT' : update ? 'UPDATE' : 'CREATE',
+    ResourcesToImport: options.resourcesToImport,
     Description: `CDK Changeset for execution ${executionId}`,
     TemplateBody: bodyParameter.TemplateBody,
     TemplateURL: bodyParameter.TemplateURL,
@@ -386,15 +407,17 @@ async function makeBodyParameter(
   resolvedEnvironment: cxapi.Environment,
   assetManifest: AssetManifestBuilder,
   toolkitInfo: ToolkitInfo,
-  sdk: ISDK): Promise<TemplateBodyParameter> {
+  sdk: ISDK,
+  overrideTemplate?: any,
+): Promise<TemplateBodyParameter> {
 
   // If the template has already been uploaded to S3, just use it from there.
-  if (stack.stackTemplateAssetObjectUrl) {
+  if (stack.stackTemplateAssetObjectUrl && !overrideTemplate) {
     return { TemplateURL: restUrlFromManifest(stack.stackTemplateAssetObjectUrl, resolvedEnvironment, sdk) };
   }
 
   // Otherwise, pass via API call (if small) or upload here (if large)
-  const templateJson = toYAML(stack.template);
+  const templateJson = toYAML(overrideTemplate ?? stack.template);
 
   if (templateJson.length <= LARGE_TEMPLATE_SIZE_KB * 1024) {
     return { TemplateBody: templateJson };
@@ -413,8 +436,15 @@ async function makeBodyParameter(
   const templateHash = contentHash(templateJson);
   const key = `cdk/${stack.id}/${templateHash}.yml`;
 
+  let templateFile = stack.templateFile;
+  if (overrideTemplate) {
+    // Add a variant of this template
+    templateFile = `${stack.templateFile}-${templateHash}.yaml`;
+    await fs.writeFile(templateFile, templateJson, { encoding: 'utf-8' });
+  }
+
   assetManifest.addFileAsset(templateHash, {
-    path: stack.templateFile,
+    path: templateFile,
   }, {
     bucketName: toolkitInfo.bucketName,
     objectKey: key,
@@ -425,6 +455,33 @@ async function makeBodyParameter(
   return { TemplateURL: templateURL };
 }
 
+/**
+ * Prepare a body parameter for CFN, performing the upload
+ *
+ * Return it as-is if it is small enough to pass in the API call,
+ * upload to S3 and return the coordinates if it is not.
+ */
+export async function makeBodyParameterAndUpload(
+  stack: cxapi.CloudFormationStackArtifact,
+  resolvedEnvironment: cxapi.Environment,
+  toolkitInfo: ToolkitInfo,
+  sdkProvider: SdkProvider,
+  sdk: ISDK,
+  overrideTemplate?: any): Promise<TemplateBodyParameter> {
+
+  // We don't have access to the actual asset manifest here, so pretend that the
+  // stack doesn't have a pre-published URL.
+  const forceUploadStack = Object.create(stack, {
+    stackTemplateAssetObjectUrl: { value: undefined },
+  });
+
+  const builder = new AssetManifestBuilder();
+  const bodyparam = await makeBodyParameter(forceUploadStack, resolvedEnvironment, builder, toolkitInfo, sdk, overrideTemplate);
+  const manifest = builder.toManifest(stack.assembly.directory);
+  await publishAssets(manifest, sdkProvider, resolvedEnvironment, { quiet: true });
+  return bodyparam;
+}
+
 export interface DestroyStackOptions {
   /**
    * The stack to be destroyed

packages/aws-cdk/lib/api/util/cloudformation.ts

@@ -16,6 +16,10 @@ interface TemplateParameter {
   [key: string]: any;
 }
 
+export type ResourceIdentifierProperties = CloudFormation.ResourceIdentifierProperties;
+export type ResourceIdentifierSummaries = CloudFormation.ResourceIdentifierSummaries;
+export type ResourcesToImport = CloudFormation.ResourcesToImport;
+
 /**
  * Represents an (existing) Stack in CloudFormation
  *