@@ -10,11 +10,11 @@ is a platform that enables you to programmatically author, schedule, and monitor
10
10
you can build a workflow for SageMaker training, hyperparameter tuning, batch transform and endpoint deployment.
11
11
You can use any SageMaker deep learning framework or Amazon algorithms to perform above operations in Airflow.
12
12
13
- There are two ways to build SageMaker workflow. Using Airflow SageMaker operators or using Airflow PythonOperator.
13
+ There are two ways to build a SageMaker workflow. Using Airflow SageMaker operators or using Airflow PythonOperator.
14
14
15
- 1. SageMaker Operators: Since Airflow 1.10.1, we contributed special operators just for SageMaker operations.
16
- Each operator takes a configuration dictionary that defines the corresponding operation. And we provide APIs to
17
- generate the configuration dictionary in SageMaker Python SDK. Current supported SageMaker operators are:
15
+ 1. SageMaker Operators: In Airflow 1.10.1, the SageMaker team contributed special operators for SageMaker operations.
16
+ Each operator takes a configuration dictionary that defines the corresponding operation. We provide APIs to generate
17
+ the configuration dictionary in the SageMaker Python SDK. Currently, the following SageMaker operators are supported :
18
18
19
19
* ``SageMakerTrainingOperator ``
20
20
* ``SageMakerTuningOperator ``
@@ -23,20 +23,20 @@ generate the configuration dictionary in SageMaker Python SDK. Current supported
23
23
* ``SageMakerEndpointConfigOperator ``
24
24
* ``SageMakerEndpointOperator ``
25
25
26
- 2. PythonOperator: Airflow built-in operator that could execute Python callables. You could use SageMaker Python SDK to
27
- customize your workflow with PythonOperator .
26
+ 2. PythonOperator: Airflow built-in operator that executes Python callables. You can use the PythonOperator to execute
27
+ operations in the SageMaker Python SDK to create a SageMaker workflow .
28
28
29
29
Using Airflow on AWS
30
30
~~~~~~~~~~~~~~~~~~~~
31
31
32
- Turbine is an open source AWS CloudFormation template to create Airflow resources stack on AWS.
32
+ Turbine is an open- source AWS CloudFormation template that enables you to create an Airflow resource stack on AWS.
33
33
You can get it here: https://github.com/villasv/aws-airflow-stack
34
34
35
35
Using Airflow SageMaker Operators
36
36
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
37
37
38
- Starting from Airflow 1.10.1, you could use SageMaker operators in Airflow. All SageMaker operators take a
39
- configuration dictionary that can be easily generated by SageMaker Python SDK. For example:
38
+ Starting with Airflow 1.10.1, you can use SageMaker operators in Airflow. All SageMaker operators take a configuration
39
+ dictionary that can be generated by the SageMaker Python SDK. For example:
40
40
41
41
.. code :: python
42
42
@@ -63,7 +63,7 @@ configuration dictionary that can be easily generated by SageMaker Python SDK. F
63
63
data = your_transform_data_s3_uri,
64
64
content_type = ' text/csv' )
65
65
66
- Now we can pass these configurations to related SageMaker operators and create the workflow:
66
+ Now you can pass these configurations to the corresponding SageMaker operators and create the workflow:
67
67
68
68
.. code :: python
69
69
@@ -99,8 +99,8 @@ Using Airflow Python Operator
99
99
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
100
100
101
101
`Airflow PythonOperator <https://airflow.apache.org/howto/operator.html?#pythonoperator >`_
102
- is a built-in operator that can execute any Python callables . If you want to build the SageMaker workflow in a more
103
- flexible way, you could write your python callables for SageMaker operations using SageMaker Python SDK. For example:
102
+ is a built-in operator that can execute any Python callable . If you want to build the SageMaker workflow in a more
103
+ flexible way, write your python callables for SageMaker operations by using the SageMaker Python SDK.
104
104
105
105
.. code :: python
106
106
@@ -125,7 +125,7 @@ flexible way, you could write your python callables for SageMaker operations us
125
125
transformer = estimator.transformer(instance_count = 1 , instance_type = ' ml.c4.xlarge' )
126
126
transformer.transform(data, content_type = ' text/csv' )
127
127
128
- Then you could build your workflow using PythonOperator with Python callables defined above:
128
+ Then build your workflow by using the PythonOperator with the Python callables defined above:
129
129
130
130
.. code :: python
131
131
@@ -158,5 +158,5 @@ Then you could build your workflow using PythonOperator with Python callables de
158
158
159
159
transform_op.set_upstream(train_op)
160
160
161
- A workflow with SageMaker training and batch transform is finished! In this way, you could customize your Python
162
- callables with SageMaker Python SDK according to your needs and build more flexible and powerful workflow .
161
+ A workflow that runs a SageMaker training job and a batch transform job is finished. You can customize your Python
162
+ callables with the SageMaker Python SDK according to your needs, and build more flexible and powerful workflows .
0 commit comments