Skip to content

Commit 1f639c7

Browse files
AmplifiyerAmplifiyer
authored
fix(cli): report errors from resource failures in nested stacks (#27318)
fix(cli): report errors from resource failures in nested stacks. ## Description Currently `StackActivityMonitor` uses `readNewEvents()` method to constantly poll CFN to get the latest deployment updates. However it only does it for the root stack and the resources in the root stack. If one of the resource in the root stack is another nested stack and one of the resource in that nested stack fails, CFN does not propagate or copy the error message in the nested stack failure rather it's a generic `Embedded stack <stackArn> was not successfully updated` This PR updates the `readNewEvents()` to recursively poll for events from the nested stack deployments as well. If errors are detected in the nested stack events, they are added to both `StackActivityMonitor:errors` as well as added in the `Printer`. Following is a before/after this change. We are deploying RootStack -> Nested Stack -> AppSync Resolver and the AppSync resolver fails with the error `Only one resolver is allowed per field` in CFN. ### Before this change ``` ✨ Synthesis time: 3.8s amplify-sample-samsara-app-pravgupt-sandbox: deploying... [1/1] amplify-sample-samsara-app-pravgupt-sandbox: creating CloudFormation changeset... 7:59:45 PM | UPDATE_FAILED | AWS::CloudFormation::Stack | data7552DF31 Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. ❌ amplify-sample-samsara-app-pravgupt-sandbox failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32) at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21 ❌ Deployment failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32) at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21 The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. ``` ### After this change ``` ✨ Synthesis time: 4.17s amplify-sample-samsara-app-pravgupt-sandbox: deploying... [1/1] amplify-sample-samsara-app-pravgupt-sandbox: creating CloudFormation changeset... 12:57:07 PM | CREATE_FAILED | AWS::AppSync::Resolver | amplifyDataL2Graph...teresolver2355E3CF Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null) 12:57:10 PM | UPDATE_FAILED | AWS::CloudFormation::Stack | data7552DF31 Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8ef d5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicater esolver2355E3CF]. ❌ amplify-sample-samsara-app-pravgupt-sandbox failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32) at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21 ❌ Deployment failed: Error: The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. at FullCloudFormationDeployment.monitorDeployment (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.js:239:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.deployStack (/Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.js:210:32) at async /Users/pravgupt/Workspaces/samsara/aws-cdk-form/aws-cdk/packages/aws-cdk/lib/util/work-graph.js:88:21 The stack named amplify-sample-samsara-app-pravgupt-sandbox failed to deploy: UPDATE_ROLLBACK_COMPLETE: Only one resolver is allowed per field. (Service: AWSAppSync; Status Code: 400; Error Code: BadRequestException; Request ID: 6f50892d-6eb4-4d19-993d-d4ed1db1ad48; Proxy: null), Embedded stack arn:aws:cloudformation:us-west-2:504152962427:stack/amplify-sample-samsara-app-pravgupt-sandbox-data7552DF31-E1S7MWRMXED4/4e1638a0-477c-11ee-a3d3-0647c8efd5b9 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [amplifyDataL2GraphqlApimyduplicateresolver2355E3CF]. ``` ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent a1f2ec2 commit 1f639c7

File tree

2 files changed

+194
-92
lines changed

2 files changed

+194
-92
lines changed

‎packages/aws-cdk/lib/api/util/cloudformation/stack-activity-monitor.ts

+13-6
Original file line numberDiff line numberDiff line change
@@ -221,14 +221,15 @@ export class StackActivityMonitor {
221221
* see a next page and the last event in the page is new to us (and within the time window).
222222
* haven't seen the final event
223223
*/
224-
private async readNewEvents(): Promise<void> {
224+
private async readNewEvents(stackName?: string): Promise<void> {
225+
const stackToPollForEvents = stackName ?? this.stackName;
225226
const events: StackActivity[] = [];
226-
227+
const CFN_SUCCESS_STATUS = ['UPDATE_COMPLETE', 'CREATE_COMPLETE', 'DELETE_COMPLETE', 'DELETE_SKIPPED'];
227228
try {
228229
let nextToken: string | undefined;
229230
let finished = false;
230231
while (!finished) {
231-
const response = await this.cfn.describeStackEvents({ StackName: this.stackName, NextToken: nextToken }).promise();
232+
const response = await this.cfn.describeStackEvents({ StackName: stackToPollForEvents, NextToken: nextToken }).promise();
232233
const eventPage = response?.StackEvents ?? [];
233234

234235
for (const event of eventPage) {
@@ -249,6 +250,13 @@ export class StackActivityMonitor {
249250
event: event,
250251
metadata: this.findMetadataFor(event.LogicalResourceId),
251252
});
253+
254+
if (event.ResourceType === 'AWS::CloudFormation::Stack' && !CFN_SUCCESS_STATUS.includes(event.ResourceStatus ?? '')) {
255+
// If the event is not for `this` stack, recursively call for events in the nested stack
256+
if (event.PhysicalResourceId !== stackToPollForEvents) {
257+
await this.readNewEvents(event.PhysicalResourceId);
258+
}
259+
}
252260
}
253261

254262
// We're also done if there's nothing left to read
@@ -258,7 +266,7 @@ export class StackActivityMonitor {
258266
}
259267
}
260268
} catch (e: any) {
261-
if (e.code === 'ValidationError' && e.message === `Stack [${this.stackName}] does not exist`) {
269+
if (e.code === 'ValidationError' && e.message === `Stack [${stackToPollForEvents}] does not exist`) {
262270
return;
263271
}
264272
throw e;
@@ -475,7 +483,7 @@ abstract class ActivityPrinterBase implements IActivityPrinter {
475483
this.resourcesPrevCompleteState[activity.event.LogicalResourceId] = status;
476484
}
477485

478-
if (hookStatus!== undefined && hookStatus.endsWith('_COMPLETE_FAILED') && activity.event.LogicalResourceId !== undefined && hookType !== undefined) {
486+
if (hookStatus !== undefined && hookStatus.endsWith('_COMPLETE_FAILED') && activity.event.LogicalResourceId !== undefined && hookType !== undefined) {
479487

480488
if (this.hookFailureMap.has(activity.event.LogicalResourceId)) {
481489
this.hookFailureMap.get(activity.event.LogicalResourceId)?.set(hookType, activity.event.HookStatusReason ?? '');
@@ -803,4 +811,3 @@ function shorten(maxWidth: number, p: string) {
803811

804812
const TIMESTAMP_WIDTH = 12;
805813
const STATUS_WIDTH = 20;
806-

‎packages/aws-cdk/test/util/stack-monitor.test.ts

+181-86
Original file line numberDiff line numberDiff line change
@@ -10,95 +10,171 @@ beforeEach(() => {
1010
printer = new FakePrinter();
1111
});
1212

13-
test('continue to the next page if it exists', async () => {
14-
await testMonitorWithEventCalls([
15-
(request) => {
16-
expect(request.NextToken).toBeUndefined();
17-
return {
18-
StackEvents: [event(102)],
19-
NextToken: 'some-token',
20-
};
21-
},
22-
(request) => {
23-
expect(request.NextToken).toBe('some-token');
24-
return {
25-
StackEvents: [event(101)],
26-
};
27-
},
28-
]);
29-
30-
// Printer sees them in chronological order
31-
expect(printer.eventIds).toEqual(['101', '102']);
32-
});
13+
describe('stack monitor event ordering and pagination', () => {
14+
test('continue to the next page if it exists', async () => {
15+
await testMonitorWithEventCalls([
16+
(request) => {
17+
expect(request.NextToken).toBeUndefined();
18+
return {
19+
StackEvents: [event(102)],
20+
NextToken: 'some-token',
21+
};
22+
},
23+
(request) => {
24+
expect(request.NextToken).toBe('some-token');
25+
return {
26+
StackEvents: [event(101)],
27+
};
28+
},
29+
]);
3330

34-
test('do not page further if we already saw the last event', async () => {
35-
await testMonitorWithEventCalls([
36-
(request) => {
37-
expect(request.NextToken).toBeUndefined();
38-
return {
39-
StackEvents: [event(101)],
40-
};
41-
},
42-
(request) => {
43-
expect(request.NextToken).toBeUndefined();
44-
return {
45-
StackEvents: [event(102), event(101)],
46-
NextToken: 'some-token',
47-
};
48-
},
49-
(request) => {
50-
// Did not use the token
51-
expect(request.NextToken).toBeUndefined();
52-
return {};
53-
},
54-
]);
55-
56-
// Seen in chronological order
57-
expect(printer.eventIds).toEqual(['101', '102']);
58-
});
31+
// Printer sees them in chronological order
32+
expect(printer.eventIds).toEqual(['101', '102']);
33+
});
34+
35+
test('do not page further if we already saw the last event', async () => {
36+
await testMonitorWithEventCalls([
37+
(request) => {
38+
expect(request.NextToken).toBeUndefined();
39+
return {
40+
StackEvents: [event(101)],
41+
};
42+
},
43+
(request) => {
44+
expect(request.NextToken).toBeUndefined();
45+
return {
46+
StackEvents: [event(102), event(101)],
47+
NextToken: 'some-token',
48+
};
49+
},
50+
(request) => {
51+
// Did not use the token
52+
expect(request.NextToken).toBeUndefined();
53+
return {};
54+
},
55+
]);
56+
57+
// Seen in chronological order
58+
expect(printer.eventIds).toEqual(['101', '102']);
59+
});
60+
61+
test('do not page further if the last event is too old', async () => {
62+
await testMonitorWithEventCalls([
63+
(request) => {
64+
expect(request.NextToken).toBeUndefined();
65+
return {
66+
StackEvents: [event(101), event(95)],
67+
NextToken: 'some-token',
68+
};
69+
},
70+
(request) => {
71+
// Start again from the top
72+
expect(request.NextToken).toBeUndefined();
73+
return {};
74+
},
75+
]);
76+
77+
// Seen only the new one
78+
expect(printer.eventIds).toEqual(['101']);
79+
});
80+
81+
test('do a final request after the monitor is stopped', async () => {
82+
await testMonitorWithEventCalls([
83+
// Before stop
84+
(request) => {
85+
expect(request.NextToken).toBeUndefined();
86+
return {
87+
StackEvents: [event(101)],
88+
};
89+
},
90+
],
91+
// After stop
92+
[
93+
(request) => {
94+
expect(request.NextToken).toBeUndefined();
95+
return {
96+
StackEvents: [event(102), event(101)],
97+
};
98+
},
99+
]);
59100

60-
test('do not page further if the last event is too old', async () => {
61-
await testMonitorWithEventCalls([
62-
(request) => {
63-
expect(request.NextToken).toBeUndefined();
64-
return {
65-
StackEvents: [event(101), event(95)],
66-
NextToken: 'some-token',
67-
};
68-
},
69-
(request) => {
70-
// Start again from the top
71-
expect(request.NextToken).toBeUndefined();
72-
return {};
73-
},
74-
]);
75-
76-
// Seen only the new one
77-
expect(printer.eventIds).toEqual(['101']);
101+
// Seen both
102+
expect(printer.eventIds).toEqual(['101', '102']);
103+
});
78104
});
79105

80-
test('do a final request after the monitor is stopped', async () => {
81-
await testMonitorWithEventCalls([
82-
// Before stop
83-
(request) => {
84-
expect(request.NextToken).toBeUndefined();
85-
return {
86-
StackEvents: [event(101)],
87-
};
88-
},
89-
],
90-
// After stop
91-
[
92-
(request) => {
93-
expect(request.NextToken).toBeUndefined();
94-
return {
95-
StackEvents: [event(102), event(101)],
96-
};
97-
},
98-
]);
99-
100-
// Seen both
101-
expect(printer.eventIds).toEqual(['101', '102']);
106+
describe('stack monitor, collecting errors from events', () => {
107+
test('return errors from the root stack', async () => {
108+
const monitor = await testMonitorWithEventCalls([
109+
(request) => {
110+
expect(request.NextToken).toBeUndefined();
111+
return {
112+
StackEvents: [addErrorToStackEvent(event(100))],
113+
};
114+
},
115+
]);
116+
117+
expect(monitor.errors).toStrictEqual(['Test Error']);
118+
});
119+
120+
test('return errors from the nested stack', async () => {
121+
const monitor = await testMonitorWithEventCalls([
122+
(request) => {
123+
expect(request.StackName).toStrictEqual('StackName');
124+
return {
125+
StackEvents: [
126+
addErrorToStackEvent(
127+
event(100), {
128+
logicalResourceId: 'nestedStackLogicalResourceId',
129+
physicalResourceId: 'nestedStackPhysicalResourceId',
130+
resourceType: 'AWS::CloudFormation::Stack',
131+
resourceStatusReason: 'nested stack failed',
132+
},
133+
),
134+
],
135+
};
136+
},
137+
(request) => {
138+
expect(request.StackName).toStrictEqual('nestedStackPhysicalResourceId');
139+
return {
140+
StackEvents: [
141+
addErrorToStackEvent(
142+
event(101), {
143+
logicalResourceId: 'nestedResource',
144+
resourceType: 'Some::Nested::Resource',
145+
resourceStatusReason: 'actual failure error message',
146+
},
147+
),
148+
],
149+
};
150+
},
151+
]);
152+
153+
expect(monitor.errors).toStrictEqual(['actual failure error message', 'nested stack failed']);
154+
});
155+
156+
test('does not check for nested stacks that have already completed successfully', async () => {
157+
const monitor = await testMonitorWithEventCalls([
158+
(request) => {
159+
expect(request.StackName).toStrictEqual('StackName');
160+
return {
161+
StackEvents: [
162+
addErrorToStackEvent(
163+
event(100), {
164+
logicalResourceId: 'nestedStackLogicalResourceId',
165+
physicalResourceId: 'nestedStackPhysicalResourceId',
166+
resourceType: 'AWS::CloudFormation::Stack',
167+
resourceStatusReason: 'nested stack status reason',
168+
resourceStatus: 'CREATE_COMPLETE',
169+
},
170+
),
171+
],
172+
};
173+
},
174+
]);
175+
176+
expect(monitor.errors).toStrictEqual([]);
177+
});
102178
});
103179

104180
const T0 = 1597837230504;
@@ -115,10 +191,28 @@ function event(nr: number): AWS.CloudFormation.StackEvent {
115191
};
116192
}
117193

194+
function addErrorToStackEvent(
195+
eventToUpdate: AWS.CloudFormation.StackEvent,
196+
props: {
197+
resourceStatus?: string,
198+
resourceType?: string,
199+
resourceStatusReason?: string,
200+
logicalResourceId?: string,
201+
physicalResourceId?: string,
202+
} = {},
203+
): AWS.CloudFormation.StackEvent {
204+
eventToUpdate.ResourceStatus = props.resourceStatus ?? 'UPDATE_FAILED';
205+
eventToUpdate.ResourceType = props.resourceType ?? 'Test::Resource::Type';
206+
eventToUpdate.ResourceStatusReason = props.resourceStatusReason ?? 'Test Error';
207+
eventToUpdate.LogicalResourceId = props.logicalResourceId ?? 'testLogicalId';
208+
eventToUpdate.PhysicalResourceId = props.physicalResourceId ?? 'testPhysicalResourceId';
209+
return eventToUpdate;
210+
}
211+
118212
async function testMonitorWithEventCalls(
119213
beforeStopInvocations: Array<(x: AWS.CloudFormation.DescribeStackEventsInput) => AWS.CloudFormation.DescribeStackEventsOutput>,
120214
afterStopInvocations: Array<(x: AWS.CloudFormation.DescribeStackEventsInput) => AWS.CloudFormation.DescribeStackEventsOutput> = [],
121-
) {
215+
): Promise<StackActivityMonitor> {
122216
let describeStackEvents = (jest.fn() as jest.Mock<AWS.CloudFormation.DescribeStackEventsOutput, [AWS.CloudFormation.DescribeStackEventsInput]>);
123217

124218
let finished = false;
@@ -144,6 +238,7 @@ async function testMonitorWithEventCalls(
144238
const monitor = new StackActivityMonitor(sdk.cloudFormation(), 'StackName', printer, undefined, new Date(T100)).start();
145239
await waitForCondition(() => finished);
146240
await monitor.stop();
241+
return monitor;
147242
}
148243

149244
class FakePrinter implements IActivityPrinter {

0 commit comments

Comments
 (0)