You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Instead of a single script file, you can submit an entire folder of code for a processing job, and optionally specify additional dependencies to be installed at job start-up by including a `requirements.txt` file.
66
+
67
+
To do this, also specify the ``source_dir`` parameter in ``.run()`` calls for :class:`SKLearnProcessor` or any other :class:`FrameworkProcessor`-based processor:
68
+
69
+
.. code:: python
70
+
71
+
sklearn_processor.run(
72
+
code="run.py", # 'processing/run.py' is the main script to run
73
+
source_dir="processing", # Upload the whole contents of 'processing/'
74
+
75
+
# If 'processing/requirements.txt' exists, the dependencies it specifies
76
+
# will be automatically installed before 'run.py' is started.
77
+
inputs=[...],
78
+
outputs=[...],
79
+
)
80
+
65
81
For an in-depth look, please see the `Scikit-learn Data Processing and Model Evaluation`_ example notebook.
66
82
67
83
.. _Scikit-learn Data Processing and Model Evaluation: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_processing/scikit_learn_data_processing_and_model_evaluation/scikit_learn_data_processing_and_model_evaluation.ipynb
@@ -220,6 +236,13 @@ For an in-depth look, please see the `Feature Transformation with Spark`_ exampl
220
236
221
237
.. _Feature Transformation with Spark: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_processing/feature_transformation_with_sagemaker_processing/feature_transformation_with_sagemaker_processing.ipynb
222
238
239
+
Data Processing with Other Frameworks
240
+
=====================================
241
+
242
+
:class:`FrameworkProcessor`-based classes are also provided for a range of other ML frameworks: For example PyTorch, TensorFlow, and MXNet.
243
+
244
+
You can use these to run data processing jobs in pre-built container environments, similarly to model training with :class:`Framework`-based Estimators.
245
+
223
246
224
247
Learn More
225
248
==========
@@ -229,12 +252,18 @@ Processing class documentation
0 commit comments