chore: hide diffs of mangled unicode strings (#25912)

I am reopening this from #25525

and following up on my comments here:
#24557 (comment)
#24557 (comment)
#25008 (comment)
#25008 (comment)
#25008 (comment)
#25008 (comment)
#25008 (comment)
#25008 (comment)
#25525 (comment)
#25525 (comment)
🫠 #25525 (comment) 🫠

---

Fixes #25309
Fixes #22203
Fixes #20212
Fixes #13634
Fixes #10523
Fixes #10219
See also: aws-cloudformation/cloudformation-coverage-roadmap#1220
See also: aws-cloudformation/cloudformation-coverage-roadmap#814

---

👻 I have retitled this PR as a `chore` instead of a `fix` because @aws-cdk-automation keeps closing my PRs as abandoned even though they are clearly not abandoned.

> This PR has been deemed to be abandoned, and will be automatically closed. Please create a new PR for these changes if you think this decision has been made in error.

---

@otaviomacedo @rix0rrr @TheRealAmazonKendra - I'm happy to adjust the approach, add more tests, or do what else needs to be done. I'm not getting any feedback from the team so I'm not sure how to proceed. The diff noise with non-ASCII information in cdk diff makes it difficult to find meaningful changes to our stacks.

🗿🗞️📬 **Crucially, this change only affects the CLI output and therefore an integration test isn't possible.**

---

CloudFormation's `GetStackTemplate` irrecoverably mangles any character not in the 7-bit ASCII range. This causes noisy output from `cdk diff` when a template contains non-English languages or emoji. We can detect this case and consider these strings equal.

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

Many AWS services accept non-ASCII input, eg many "description" fields. CloudFormation will correctly dispatch these templates but when invoking `GetStackTemplate` the result is mangled. This causes annoying noise in the output of `cdk diff`:

```
Resources
[~] AWS::Lambda::Function Lambda/Resource
 └─ [~] Description
     ├─ [-] ?????
     └─ [+] 🤦🏻‍♂️
```

This change modifies the diff algorithm to consider the string equal if the lvalue is a mangled version of the rvalue.

Of course this runs the risk of hiding changesets which modify only a single non-ASCII character to another non-ASCII character, but these fields already tend to be informative in nature.

Author: laverdet

packages/@aws-cdk/cloudformation-diff/lib/diff/util.ts

@@ -134,6 +134,18 @@ export function unionOf(lv: string[] | Set<string>, rv: string[] | Set<string>):
   return new Array(...result);
 }
 
+/**
+ * GetStackTemplate flattens any codepoint greater than "\u7f" to "?". This is
+ * true even for codepoints in the supplemental planes which are represented
+ * in JS as surrogate pairs, all the way up to "\u{10ffff}".
+ *
+ * This function implements the same mangling in order to provide diagnostic
+ * information in `cdk diff`.
+ */
+export function mangleLikeCloudFormation(payload: string) {
+  return payload.replace(/[\u{80}-\u{10ffff}]/gu, '?');
+}
+
 /**
  * A parseFloat implementation that does the right thing for
  * strings like '0.0.0'

packages/@aws-cdk/cloudformation-diff/lib/index.ts

@@ -1,4 +1,4 @@
 export * from './diff-template';
 export * from './format';
 export * from './format-table';
-export { deepEqual } from './diff/util';
+export { deepEqual, mangleLikeCloudFormation } from './diff/util';

packages/@aws-cdk/cloudformation-diff/test/util-test.ts

@@ -0,0 +1,10 @@
+import { mangleLikeCloudFormation } from '../lib/diff/util';
+
+test('mangled strings', () => {
+  expect(mangleLikeCloudFormation('foo')).toEqual('foo');
+  expect(mangleLikeCloudFormation('文字化け')).toEqual('????');
+  expect(mangleLikeCloudFormation('🤦🏻‍♂️')).toEqual('?????');
+  expect(mangleLikeCloudFormation('\u{10ffff}')).toEqual('?');
+  expect(mangleLikeCloudFormation('\u007f')).toEqual('\u007f');
+  expect(mangleLikeCloudFormation('\u0080')).toEqual('?');
+});

packages/aws-cdk/lib/cli.ts

@@ -257,7 +257,7 @@ async function parseCommandLineArguments(args: string[]) {
       .option('exclusively', { type: 'boolean', alias: 'e', desc: 'Only diff requested stacks, don\'t include dependencies' })
       .option('context-lines', { type: 'number', desc: 'Number of context lines to include in arbitrary JSON diff rendering', default: 3, requiresArg: true })
       .option('template', { type: 'string', desc: 'The path to the CloudFormation template to compare with', requiresArg: true })
-      .option('strict', { type: 'boolean', desc: 'Do not filter out AWS::CDK::Metadata resources', default: false })
+      .option('strict', { type: 'boolean', desc: 'Do not filter out AWS::CDK::Metadata resources or mangled non-ASCII characters', default: false })
       .option('security-only', { type: 'boolean', desc: 'Only diff for broadened security changes', default: false })
       .option('fail', { type: 'boolean', desc: 'Fail with exit code 1 in case of diff' })
       .option('processed', { type: 'boolean', desc: 'Whether to compare against the template with Transforms already processed', default: false }))

packages/aws-cdk/lib/diff.ts

@@ -21,7 +21,18 @@ export function printStackDiff(
   context: number,
   stream?: cfnDiff.FormatStream): number {
 
-  const diff = cfnDiff.diffTemplate(oldTemplate, newTemplate.template);
+  let diff = cfnDiff.diffTemplate(oldTemplate, newTemplate.template);
+
+  // detect and filter out mangled characters from the diff
+  let filteredChangesCount = 0;
+  if (diff.differenceCount && !strict) {
+    const mangledNewTemplate = JSON.parse(cfnDiff.mangleLikeCloudFormation(JSON.stringify(newTemplate.template)));
+    const mangledDiff = cfnDiff.diffTemplate(oldTemplate, mangledNewTemplate);
+    filteredChangesCount = Math.max(0, diff.differenceCount - mangledDiff.differenceCount);
+    if (filteredChangesCount > 0) {
+      diff = mangledDiff;
+    }
+  }
 
   // filter out 'AWS::CDK::Metadata' resources from the template
   if (diff.resources && !strict) {
@@ -41,6 +52,9 @@ export function printStackDiff(
   } else {
     print(chalk.green('There were no differences'));
   }
+  if (filteredChangesCount > 0) {
+    print(chalk.yellow(`Omitted ${filteredChangesCount} changes because they are likely mangled non-ASCII characters. Use --strict to print them.`));
+  }
 
   return diff.differenceCount;
 }