Skip to content

Commit 5592553

Browse files
badmintoncryerlpizzinidevGavinZZ
authored
feat(stepfunctions-tasks): start glue crawler (#29016)
### Issue Closes #24188. ### Reason for this change AWS Step Functions supports the ability to start a Glue Crawler as a task within a state machine. However, this is not configurable. ### Description of changes I've implemented `GlueStartCrawlerRun` class in stepfunctions-tasks and we can create start crawler task easily: ```ts const task = new GlueStartCrawlerRun(stack, 'Task', { crawlerName: 'glue-crawler-name', }); ``` ### Description of how you validated changes I've added both unit and integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license* --------- Co-authored-by: Luca Pizzini <[email protected]> Co-authored-by: GZ <[email protected]>
1 parent 840ec97 commit 5592553

File tree

13 files changed

+1089
-0
lines changed

13 files changed

+1089
-0
lines changed

packages/@aws-cdk-testing/framework-integ/test/aws-stepfunctions-tasks/test/glue/integ.glue-start-crawler.js.snapshot/EcsDeploymentConfigTestDefaultTestDeployAssertA6573788.assets.json

+19
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-stepfunctions-tasks/test/glue/integ.glue-start-crawler.js.snapshot/EcsDeploymentConfigTestDefaultTestDeployAssertA6573788.template.json

+36
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-stepfunctions-tasks/test/glue/integ.glue-start-crawler.js.snapshot/aws-cdk-glue-crawler.assets.json

+19
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
{
2+
"Resources": {
3+
"Bucket83908E77": {
4+
"Type": "AWS::S3::Bucket",
5+
"UpdateReplacePolicy": "Delete",
6+
"DeletionPolicy": "Delete"
7+
},
8+
"GluecrawlwerRole4E24839F": {
9+
"Type": "AWS::IAM::Role",
10+
"Properties": {
11+
"AssumeRolePolicyDocument": {
12+
"Statement": [
13+
{
14+
"Action": "sts:AssumeRole",
15+
"Effect": "Allow",
16+
"Principal": {
17+
"Service": "glue.amazonaws.com"
18+
}
19+
}
20+
],
21+
"Version": "2012-10-17"
22+
},
23+
"ManagedPolicyArns": [
24+
{
25+
"Fn::Join": [
26+
"",
27+
[
28+
"arn:",
29+
{
30+
"Ref": "AWS::Partition"
31+
},
32+
":iam::aws:policy/service-role/AWSGlueServiceRole"
33+
]
34+
]
35+
}
36+
]
37+
}
38+
},
39+
"Database": {
40+
"Type": "AWS::Glue::Database",
41+
"Properties": {
42+
"CatalogId": {
43+
"Ref": "AWS::AccountId"
44+
},
45+
"DatabaseInput": {
46+
"Description": "My database",
47+
"Name": "my-database"
48+
}
49+
}
50+
},
51+
"Crawler": {
52+
"Type": "AWS::Glue::Crawler",
53+
"Properties": {
54+
"DatabaseName": {
55+
"Ref": "Database"
56+
},
57+
"Role": {
58+
"Fn::GetAtt": [
59+
"GluecrawlwerRole4E24839F",
60+
"Arn"
61+
]
62+
},
63+
"Targets": {
64+
"S3Targets": [
65+
{
66+
"Path": {
67+
"Fn::Join": [
68+
"",
69+
[
70+
"s3://",
71+
{
72+
"Ref": "Bucket83908E77"
73+
},
74+
"/"
75+
]
76+
]
77+
}
78+
}
79+
]
80+
}
81+
}
82+
},
83+
"StateMachineRole543B9670": {
84+
"Type": "AWS::IAM::Role",
85+
"Properties": {
86+
"AssumeRolePolicyDocument": {
87+
"Statement": [
88+
{
89+
"Action": "sts:AssumeRole",
90+
"Effect": "Allow",
91+
"Principal": {
92+
"Service": "states.amazonaws.com"
93+
}
94+
}
95+
],
96+
"Version": "2012-10-17"
97+
}
98+
}
99+
},
100+
"StateMachineRoleDefaultPolicyDA5F7DA8": {
101+
"Type": "AWS::IAM::Policy",
102+
"Properties": {
103+
"PolicyDocument": {
104+
"Statement": [
105+
{
106+
"Action": [
107+
"glue:GetCrawler",
108+
"glue:StartCrawler"
109+
],
110+
"Effect": "Allow",
111+
"Resource": {
112+
"Fn::Join": [
113+
"",
114+
[
115+
"arn:",
116+
{
117+
"Ref": "AWS::Partition"
118+
},
119+
":glue:",
120+
{
121+
"Ref": "AWS::Region"
122+
},
123+
":",
124+
{
125+
"Ref": "AWS::AccountId"
126+
},
127+
":crawler/",
128+
{
129+
"Ref": "Crawler"
130+
}
131+
]
132+
]
133+
}
134+
}
135+
],
136+
"Version": "2012-10-17"
137+
},
138+
"PolicyName": "StateMachineRoleDefaultPolicyDA5F7DA8",
139+
"Roles": [
140+
{
141+
"Ref": "StateMachineRole543B9670"
142+
}
143+
]
144+
}
145+
},
146+
"StateMachine81935E76": {
147+
"Type": "AWS::StepFunctions::StateMachine",
148+
"Properties": {
149+
"DefinitionString": {
150+
"Fn::Join": [
151+
"",
152+
[
153+
"{\"StartAt\":\"Start Task\",\"States\":{\"Start Task\":{\"Type\":\"Pass\",\"Next\":\"Glue Crawler Task\"},\"Glue Crawler Task\":{\"Next\":\"End Task\",\"Type\":\"Task\",\"Resource\":\"arn:",
154+
{
155+
"Ref": "AWS::Partition"
156+
},
157+
":states:::aws-sdk:glue:startCrawler\",\"Parameters\":{\"Name\":\"",
158+
{
159+
"Ref": "Crawler"
160+
},
161+
"\"}},\"End Task\":{\"Type\":\"Pass\",\"End\":true}}}"
162+
]
163+
]
164+
},
165+
"RoleArn": {
166+
"Fn::GetAtt": [
167+
"StateMachineRole543B9670",
168+
"Arn"
169+
]
170+
}
171+
},
172+
"DependsOn": [
173+
"StateMachineRoleDefaultPolicyDA5F7DA8",
174+
"StateMachineRole543B9670"
175+
],
176+
"UpdateReplacePolicy": "Delete",
177+
"DeletionPolicy": "Delete"
178+
}
179+
},
180+
"Parameters": {
181+
"BootstrapVersion": {
182+
"Type": "AWS::SSM::Parameter::Value<String>",
183+
"Default": "/cdk-bootstrap/hnb659fds/version",
184+
"Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
185+
}
186+
},
187+
"Rules": {
188+
"CheckBootstrapVersion": {
189+
"Assertions": [
190+
{
191+
"Assert": {
192+
"Fn::Not": [
193+
{
194+
"Fn::Contains": [
195+
[
196+
"1",
197+
"2",
198+
"3",
199+
"4",
200+
"5"
201+
],
202+
{
203+
"Ref": "BootstrapVersion"
204+
}
205+
]
206+
}
207+
]
208+
},
209+
"AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
210+
}
211+
]
212+
}
213+
}
214+
}

packages/@aws-cdk-testing/framework-integ/test/aws-stepfunctions-tasks/test/glue/integ.glue-start-crawler.js.snapshot/cdk.out

+1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-stepfunctions-tasks/test/glue/integ.glue-start-crawler.js.snapshot/integ.json

+12
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)