Skip to content

doc: consolidate framework version and image information #1522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 22, 2020
Merged

doc: consolidate framework version and image information #1522

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2a34e41
doc: consolidate framework version and image information
laurenyu May 21, 2020
82b1a5a
Merge branch 'master' into point-to-dlc-docs
laurenyu May 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 25 additions & 137 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,31 +39,31 @@ For detailed API reference please go to: `Read the Docs <https://sagemaker.readt
Table of Contents
-----------------

1. `Installing SageMaker Python SDK <#installing-the-sagemaker-python-sdk>`__
2. `Using the SageMaker Python SDK <https://sagemaker.readthedocs.io/en/stable/overview.html>`__
3. `MXNet SageMaker Estimators <#mxnet-sagemaker-estimators>`__
4. `TensorFlow SageMaker Estimators <#tensorflow-sagemaker-estimators>`__
5. `Chainer SageMaker Estimators <#chainer-sagemaker-estimators>`__
6. `PyTorch SageMaker Estimators <#pytorch-sagemaker-estimators>`__
7. `Scikit-learn SageMaker Estimators <#scikit-learn-sagemaker-estimators>`__
8. `XGBoost SageMaker Estimators <#xgboost-sagemaker-estimators>`__
9. `SageMaker Reinforcement Learning Estimators <#sagemaker-reinforcement-learning-estimators>`__
10. `SageMaker SparkML Serving <#sagemaker-sparkml-serving>`__
11. `AWS SageMaker Estimators <#aws-sagemaker-estimators>`__
12. `Using SageMaker AlgorithmEstimators <https://sagemaker.readthedocs.io/en/stable/overview.html#using-sagemaker-algorithmestimators>`__
13. `Consuming SageMaker Model Packages <https://sagemaker.readthedocs.io/en/stable/overview.html#consuming-sagemaker-model-packages>`__
14. `BYO Docker Containers with SageMaker Estimators <https://sagemaker.readthedocs.io/en/stable/overview.html#byo-docker-containers-with-sagemaker-estimators>`__
15. `SageMaker Automatic Model Tuning <https://sagemaker.readthedocs.io/en/stable/overview.html#sagemaker-automatic-model-tuning>`__
16. `SageMaker Batch Transform <https://sagemaker.readthedocs.io/en/stable/overview.html#sagemaker-batch-transform>`__
17. `Secure Training and Inference with VPC <https://sagemaker.readthedocs.io/en/stable/overview.html#secure-training-and-inference-with-vpc>`__
18. `BYO Model <https://sagemaker.readthedocs.io/en/stable/overview.html#byo-model>`__
19. `Inference Pipelines <https://sagemaker.readthedocs.io/en/stable/overview.html#inference-pipelines>`__
20. `Amazon SageMaker Operators for Kubernetes <#amazon-sagemaker-operators-for-kubernetes>`__
21. `Amazon SageMaker Operators in Apache Airflow <#sagemaker-workflow>`__
22. `SageMaker Autopilot <#sagemaker-autopilot>`__
23. `Model Monitoring <#amazon-sagemaker-model-monitoring>`__
24. `SageMaker Debugger <#amazon-sagemaker-debugger>`__
25. `SageMaker Processing <#amazon-sagemaker-processing>`__
#. `Installing SageMaker Python SDK <#installing-the-sagemaker-python-sdk>`__
#. `Using the SageMaker Python SDK <https://sagemaker.readthedocs.io/en/stable/overview.html>`__
#. `Using MXNet <https://sagemaker.readthedocs.io/en/stable/using_mxnet.html>`__
#. `Using TensorFlow <https://sagemaker.readthedocs.io/en/stable/using_tf.html>`__
#. `Using Chainer <https://sagemaker.readthedocs.io/en/stable/using_chainer.html>`__
#. `Using PyTorch <https://sagemaker.readthedocs.io/en/stable/using_pytorch.html>`__
#. `Scikit-learn SageMaker Estimators <#scikit-learn-sagemaker-estimators>`__
#. `XGBoost SageMaker Estimators <#xgboost-sagemaker-estimators>`__
#. `SageMaker Reinforcement Learning Estimators <https://sagemaker.readthedocs.io/en/stable/using_rl.html>`__
#. `SageMaker SparkML Serving <#sagemaker-sparkml-serving>`__
#. `AWS SageMaker Estimators <#aws-sagemaker-estimators>`__
#. `Using SageMaker AlgorithmEstimators <https://sagemaker.readthedocs.io/en/stable/overview.html#using-sagemaker-algorithmestimators>`__
#. `Consuming SageMaker Model Packages <https://sagemaker.readthedocs.io/en/stable/overview.html#consuming-sagemaker-model-packages>`__
#. `BYO Docker Containers with SageMaker Estimators <https://sagemaker.readthedocs.io/en/stable/overview.html#byo-docker-containers-with-sagemaker-estimators>`__
#. `SageMaker Automatic Model Tuning <https://sagemaker.readthedocs.io/en/stable/overview.html#sagemaker-automatic-model-tuning>`__
#. `SageMaker Batch Transform <https://sagemaker.readthedocs.io/en/stable/overview.html#sagemaker-batch-transform>`__
#. `Secure Training and Inference with VPC <https://sagemaker.readthedocs.io/en/stable/overview.html#secure-training-and-inference-with-vpc>`__
#. `BYO Model <https://sagemaker.readthedocs.io/en/stable/overview.html#byo-model>`__
#. `Inference Pipelines <https://sagemaker.readthedocs.io/en/stable/overview.html#inference-pipelines>`__
#. `Amazon SageMaker Operators for Kubernetes <#amazon-sagemaker-operators-for-kubernetes>`__
#. `Amazon SageMaker Operators in Apache Airflow <#sagemaker-workflow>`__
#. `SageMaker Autopilot <#sagemaker-autopilot>`__
#. `Model Monitoring <https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html>`__
#. `SageMaker Debugger <https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_debugger.html>`__
#. `SageMaker Processing <https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html>`__


Installing the SageMaker Python SDK
Expand Down Expand Up @@ -197,73 +197,6 @@ Preview the site with a Python web server:

View the website by visiting http://localhost:8000


MXNet SageMaker Estimators
--------------------------

By using MXNet SageMaker Estimators, you can train and host MXNet models on Amazon SageMaker.

Supported versions of MXNet: ``0.12.1``, ``1.0.0``, ``1.1.0``, ``1.2.1``, ``1.3.0``, ``1.4.0``, ``1.4.1``, ``1.6.0``.

Supported versions of MXNet for Elastic Inference: ``1.3.0``, ``1.4.0``, ``1.4.1``, ``1.5.1``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information, see `Using MXNet with the SageMaker Python SDK`_.

.. _Using MXNet with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_mxnet.html


TensorFlow SageMaker Estimators
-------------------------------

By using TensorFlow SageMaker Estimators, you can train and host TensorFlow models on Amazon SageMaker.

Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14.0``, ``1.15.0``, ``1.15.2``, ``2.0.0``, ``2.0.1``, ``2.1.0``.

Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14.0``, ``1.15.0``, ``2.0.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information, see `Using TensorFlow with the SageMaker Python SDK`_.

.. _Using TensorFlow with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_tf.html


Chainer SageMaker Estimators
----------------------------

By using Chainer SageMaker Estimators, you can train and host Chainer models on Amazon SageMaker.

Supported versions of Chainer: ``4.0.0``, ``4.1.0``, ``5.0.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information about Chainer, see https://github.com/chainer/chainer.

For more information about Chainer SageMaker Estimators, see `Using Chainer with the SageMaker Python SDK`_.

.. _Using Chainer with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_chainer.html


PyTorch SageMaker Estimators
----------------------------

With PyTorch SageMaker Estimators, you can train and host PyTorch models on Amazon SageMaker.

Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``, ``1.4.0``, ``1.5.0``.

Supported versions of PyTorch for Elastic Inference: ``1.3.1``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information about PyTorch, see https://github.com/pytorch/pytorch.

For more information about PyTorch SageMaker Estimators, see `Using PyTorch with the SageMaker Python SDK`_.

.. _Using PyTorch with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_pytorch.html


Scikit-learn SageMaker Estimators
---------------------------------

Expand Down Expand Up @@ -295,22 +228,6 @@ For more information about XGBoost SageMaker Estimators, see `Using XGBoost with
.. _Using XGBoost with the SageMaker Python SDK: https://sagemaker.readthedocs.io/en/stable/using_xgboost.html


SageMaker Reinforcement Learning Estimators
-------------------------------------------

With Reinforcement Learning (RL) Estimators, you can use reinforcement learning to train models on Amazon SageMaker.

Supported versions of Coach: ``0.10.1``, ``0.11.1`` with TensorFlow, ``0.11.0`` with TensorFlow or MXNet.
For more information about Coach, see https://github.com/NervanaSystems/coach

Supported versions of Ray: ``0.5.3``, ``0.6.5`` with TensorFlow.
For more information about Ray, see https://github.com/ray-project/ray

For more information about SageMaker RL Estimators, see `SageMaker Reinforcement Learning Estimators`_.

.. _SageMaker Reinforcement Learning Estimators: src/sagemaker/rl/README.rst


SageMaker SparkML Serving
-------------------------

Expand Down Expand Up @@ -385,32 +302,3 @@ on your data, and hosts a series of models on an Inference Pipeline.
For more information about SageMaker Autopilot, see `SageMaker Autopilot`_.

.. _SageMaker Autopilot: src/sagemaker/automl/README.rst

Amazon SageMaker Model Monitoring
---------------------------------

You can use Amazon SageMaker Model Monitoring to automatically detect concept drift by monitoring your machine learning models.

For more information, see `Amazon SageMaker Model Monitoring`_.

.. _Amazon SageMaker Model Monitoring: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html

Amazon SageMaker Debugger
-------------------------

You can use Amazon SageMaker Debugger to automatically detect anomalies while training your machine learning models.

For more information, see `Amazon SageMaker Debugger`_.

.. _Amazon SageMaker Debugger: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_debugger.html


Amazon SageMaker Processing
---------------------------------

You can use Amazon SageMaker Processing to perform data processing tasks such as data pre- and post-processing, feature engineering, data validation, and model evaluation


For more information, see `Amazon SageMaker Processing`_.

.. _Amazon SageMaker Processing: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html
57 changes: 53 additions & 4 deletions doc/using_chainer.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,11 @@ Using Chainer with the SageMaker Python SDK

With Chainer Estimators, you can train and host Chainer models on Amazon SageMaker.

For information about supported versions of Chainer, see the `Chainer README <https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/chainer/README.rst>`__.
Supported versions of Chainer: ``4.0.0``, ``4.1.0``, ``5.0.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

For more information about Chainer, see https://github.com/chainer/chainer.

For general information about using the SageMaker Python SDK, see :ref:`overview:Using the SageMaker Python SDK`.

Expand Down Expand Up @@ -638,6 +642,51 @@ The following are optional arguments. When you create a ``Chainer`` object, you
SageMaker Chainer Docker containers
***********************************

You can visit the SageMaker Chainer containers repository here: https://github.com/aws/sagemaker-chainer-container

For information about SageMaker Chainer Docker containers and their dependencies, see `SageMaker Chainer Docker containers <https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/chainer#sagemaker-chainer-docker-containers>`_.
When training and deploying training scripts, SageMaker runs your Python script in a Docker container with several
libraries installed. When creating the Estimator and calling deploy to create the SageMaker Endpoint, you can control
the environment your script runs in.

SageMaker runs Chainer Estimator scripts in either Python 2.7 or Python 3.5. You can select the Python version by
passing a py_version keyword arg to the Chainer Estimator constructor. Setting this to py3 (the default) will cause your
training script to be run on Python 3.5. Setting this to py2 will cause your training script to be run on Python 2.7
This Python version applies to both the Training Job, created by fit, and the Endpoint, created by deploy.

The Chainer Docker images have the following dependencies installed:

+-----------------------------+-------------+-------------+-------------+
| Dependencies | chainer 4.0 | chainer 4.1 | chainer 5.0 |
+-----------------------------+-------------+-------------+-------------+
| chainer | 4.0.0 | 4.1.0 | 5.0.0 |
+-----------------------------+-------------+-------------+-------------+
| chainercv | 0.9.0 | 0.10.0 | 0.10.0 |
+-----------------------------+-------------+-------------+-------------+
| chainermn | 1.2.0 | 1.3.0 | N/A |
+-----------------------------+-------------+-------------+-------------+
| CUDA (GPU image only) | 9.0 | 9.0 | 9.0 |
+-----------------------------+-------------+-------------+-------------+
| cupy | 4.0.0 | 4.1.0 | 5.0.0 |
+-----------------------------+-------------+-------------+-------------+
| matplotlib | 2.2.0 | 2.2.0 | 2.2.0 |
+-----------------------------+-------------+-------------+-------------+
| mpi4py | 3.0.0 | 3.0.0 | 3.0.0 |
+-----------------------------+-------------+-------------+-------------+
| numpy | 1.14.3 | 1.15.3 | 1.15.4 |
+-----------------------------+-------------+-------------+-------------+
| opencv-python | 3.4.0.12 | 3.4.0.12 | 3.4.0.12 |
+-----------------------------+-------------+-------------+-------------+
| Pillow | 5.1.0 | 5.3.0 | 5.3.0 |
+-----------------------------+-------------+-------------+-------------+
| Python | 2.7 or 3.5 | 2.7 or 3.5 | 2.7 or 3.5 |
+-----------------------------+-------------+-------------+-------------+

The Docker images extend Ubuntu 16.04.

You must select a version of Chainer by passing a ``framework_version`` keyword arg to the Chainer Estimator
constructor. Currently supported versions are listed in the above table. You can also set framework_version to only
specify major and minor version, which will cause your training script to be run on the latest supported patch
version of that minor version.

Alternatively, you can build your own image by following the instructions in the SageMaker Chainer containers
repository, and passing ``image_name`` to the Chainer Estimator constructor.

You can visit the SageMaker Chainer containers repository at https://github.com/aws/sagemaker-chainer-container
14 changes: 8 additions & 6 deletions doc/using_mxnet.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ Use MXNet with the SageMaker Python SDK

With the SageMaker Python SDK, you can train and host MXNet models on Amazon SageMaker.

For information about supported versions of MXNet, see the `MXNet README <https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/mxnet/README.rst>`__.
For information about supported versions of MXNet, see the `AWS documentation <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html>`__.

We recommend that you use the latest supported version because that's where we focus our development efforts.

For general information about using the SageMaker Python SDK, see :ref:`overview:Using the SageMaker Python SDK`.

Expand Down Expand Up @@ -807,9 +809,9 @@ For information about the different MXNet-related classes in the SageMaker Pytho
SageMaker MXNet Containers
**************************

For information about SageMaker MXNet containers, see the following topics:

- training: https://github.com/aws/sagemaker-mxnet-container
- serving: https://github.com/aws/sagemaker-mxnet-serving-container
For information about the SageMaker MXNet containers, see:

For information about the dependencies installed in SageMaker MXNet containers, see https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/mxnet/README.rst#sagemaker-mxnet-containers.
- `SageMaker MXNet training toolkit <https://github.com/aws/sagemaker-mxnet-container>`_
- `SageMaker MXNet serving toolkit <https://github.com/aws/sagemaker-mxnet-serving-container>`_
- `Deep Learning Container (DLC) Dockerfiles for MXNet <https://github.com/aws/deep-learning-containers/tree/master/mxnet>`_
- `Deep Learning Container (DLC) Images <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html>`_ and `release notes <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/dlc-release-notes.html>`_
11 changes: 6 additions & 5 deletions doc/using_pytorch.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ Using PyTorch with the SageMaker Python SDK

With PyTorch Estimators and Models, you can train and host PyTorch models on Amazon SageMaker.

Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``, ``1.4.0``, ``1.5.0``.

Supported versions of PyTorch for Elastic Inference: ``1.3.1``.
For information about supported versions of PyTorch, see the `AWS documentation <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html>`__.

We recommend that you use the latest supported version because that's where we focus our development efforts.

Expand Down Expand Up @@ -758,6 +756,9 @@ The following are optional arguments. When you create a ``PyTorch`` object, you
SageMaker PyTorch Docker Containers
***********************************

For information about SageMaker PyTorch containers, see `the SageMaker PyTorch container repository <https://github.com/aws/sagemaker-pytorch-container>`_ and `SageMaker PyTorch Serving container repository <https://github.com/aws/sagemaker-pytorch-serving-container>`__.
For information about the SageMaker PyTorch containers, see:

For information about SageMaker PyTorch container dependencies, see `SageMaker PyTorch Containers <https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/pytorch#sagemaker-pytorch-docker-containers>`_.
- `SageMaker PyTorch training toolkit <https://github.com/aws/sagemaker-pytorch-container>`_
- `SageMaker PyTorch serving toolkit <https://github.com/aws/sagemaker-pytorch-serving-container>`_
- `Deep Learning Container (DLC) Dockerfiles for PyTorch <https://github.com/aws/deep-learning-containers/tree/master/pytorch>`_
- `Deep Learning Container (DLC) Images <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html>`_ and `release notes <https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/dlc-release-notes.html>`_
Loading