Skip to content

Commit 5fc171b

Browse files
committed
doc: improve FrameworkProcessor guide docs
Ensure Processor classes are listed in framework API ToCs, and add a brief introduction to usage in the SageMaker Processing guide.
1 parent 0fd7f1d commit 5fc171b

File tree

6 files changed

+68
-2
lines changed

6 files changed

+68
-2
lines changed

doc/amazon_sagemaker_processing.rst

+29
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,22 @@ Then you can run a scikit-learn script ``preprocessing.py`` in a processing job.
6262
6363
preprocessing_job_description = sklearn_processor.jobs[-1].describe()
6464
65+
Instead of a single script file, you can submit an entire folder of code for a processing job, and optionally specify additional dependencies to be installed at job start-up by including a `requirements.txt` file.
66+
67+
To do this, also specify the ``source_dir`` parameter in ``.run()`` calls for :class:`SKLearnProcessor` or any other :class:`FrameworkProcessor`-based processor:
68+
69+
.. code:: python
70+
71+
sklearn_processor.run(
72+
code="run.py", # 'processing/run.py' is the main script to run
73+
source_dir="processing", # Upload the whole contents of 'processing/'
74+
75+
# If 'processing/requirements.txt' exists, the dependencies it specifies
76+
# will be automatically installed before 'run.py' is started.
77+
inputs=[...],
78+
outputs=[...],
79+
)
80+
6581
For an in-depth look, please see the `Scikit-learn Data Processing and Model Evaluation`_ example notebook.
6682

6783
.. _Scikit-learn Data Processing and Model Evaluation: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_processing/scikit_learn_data_processing_and_model_evaluation/scikit_learn_data_processing_and_model_evaluation.ipynb
@@ -220,6 +236,13 @@ For an in-depth look, please see the `Feature Transformation with Spark`_ exampl
220236

221237
.. _Feature Transformation with Spark: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_processing/feature_transformation_with_sagemaker_processing/feature_transformation_with_sagemaker_processing.ipynb
222238

239+
Data Processing with Other Frameworks
240+
=====================================
241+
242+
:class:`FrameworkProcessor`-based classes are also provided for a range of other ML frameworks: For example PyTorch, TensorFlow, and MXNet.
243+
244+
You can use these to run data processing jobs in pre-built container environments, similarly to model training with :class:`Framework`-based Estimators.
245+
223246

224247
Learn More
225248
==========
@@ -229,12 +252,18 @@ Processing class documentation
229252

230253
- :class:`sagemaker.processing.Processor`
231254
- :class:`sagemaker.processing.ScriptProcessor`
255+
- :class:`sagemaker.processing.FrameworkProcessor`
232256
- :class:`sagemaker.sklearn.processing.SKLearnProcessor`
233257
- :class:`sagemaker.spark.processing.PySparkProcessor`
234258
- :class:`sagemaker.spark.processing.SparkJarProcessor`
235259
- :class:`sagemaker.processing.ProcessingInput`
236260
- :class:`sagemaker.processing.ProcessingOutput`
237261
- :class:`sagemaker.processing.ProcessingJob`
262+
- :class:`sagemaker.huggingface.processing.HuggingFaceProcessor`
263+
- :class:`sagemaker.mxnet.processing.MXNetProcessor`
264+
- :class:`sagemaker.pytorch.processing.PyTorchProcessor`
265+
- :class:`sagemaker.tensorflow.processing.TensorFlowProcessor`
266+
- :class:`sagemaker.xgboost.processing.XGBoostProcessor`
238267

239268

240269
Further documentation

doc/frameworks/huggingface/sagemaker.huggingface.rst

+10-2
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,18 @@ Hugging Face Model
1717
:undoc-members:
1818
:show-inheritance:
1919

20-
HuggingFace Predictor
21-
---------------------
20+
Hugging Face Predictor
21+
----------------------
2222

2323
.. autoclass:: sagemaker.huggingface.model.HuggingFacePredictor
2424
:members:
2525
:undoc-members:
2626
:show-inheritance:
27+
28+
Hugging Face Processor
29+
----------------------
30+
31+
.. autoclass:: sagemaker.huggingface.processing.HuggingFaceProcessor
32+
:members:
33+
:undoc-members:
34+
:show-inheritance:

doc/frameworks/mxnet/sagemaker.mxnet.rst

+8
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,11 @@ MXNet Predictor
2525
:members:
2626
:undoc-members:
2727
:show-inheritance:
28+
29+
MXNet Processor
30+
---------------------------
31+
32+
.. autoclass:: sagemaker.mxnet.processing.MXNetProcessor
33+
:members:
34+
:undoc-members:
35+
:show-inheritance:

doc/frameworks/pytorch/sagemaker.pytorch.rst

+8
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,11 @@ PyTorch Predictor
2424
:members:
2525
:undoc-members:
2626
:show-inheritance:
27+
28+
PyTorch Processor
29+
-----------------
30+
31+
.. autoclass:: sagemaker.pytorch.processing.PyTorchProcessor
32+
:members:
33+
:undoc-members:
34+
:show-inheritance:

doc/frameworks/tensorflow/sagemaker.tensorflow.rst

+8
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,11 @@ TensorFlow Serving Predictor
2525
:members:
2626
:undoc-members:
2727
:show-inheritance:
28+
29+
TensorFlow Processor
30+
--------------------
31+
32+
.. autoclass:: sagemaker.tensorflow.processing.TensorFlowProcessor
33+
:members:
34+
:undoc-members:
35+
:show-inheritance:

doc/frameworks/xgboost/xgboost.rst

+5
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,8 @@ The Amazon SageMaker XGBoost open source framework algorithm.
1616
:members:
1717
:undoc-members:
1818
:show-inheritance:
19+
20+
.. autoclass:: sagemaker.xgboost.processing.XGBoostProcessor
21+
:members:
22+
:undoc-members:
23+
:show-inheritance:

0 commit comments

Comments
 (0)