Skip to content

change: upgrade framework versions and add gamma test env #1315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Feb 27, 2020
Merged

change: upgrade framework versions and add gamma test env #1315

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0496f42
change: upgrade framework versions and add gamma test env
Feb 26, 2020
aa74a40
fix: pytorch 1.4.0 py2 not supported
Feb 26, 2020
8851af6
Merge branch 'master' into framework-gamma
chuyang-deng Feb 26, 2020
254acea
newline EOF tox.ini
Feb 26, 2020
9b2c6d6
Merge branch 'framework-gamma' of github.com:ChuyangDeng/sagemaker-py…
Feb 26, 2020
3fd7355
update tf py2 error unit test
Feb 27, 2020
ea781ed
remove gamma
Feb 27, 2020
21f0a6a
update pytorch py2 test skip reason
Feb 27, 2020
dc5d407
update pytorch_train skip message
Feb 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/sagemaker/mxnet/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ The Docker images have the following dependencies installed:
+-------------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| CUDA (GPU image only) | 9.0 | 9.0 | 9.0 | 9.0 | 9.0 | 9.2 | 10.0 | 10.1 |
+-------------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| numpy | 1.13.3 | 1.13.3 | 1.13.3 | 1.14.5 | 1.14.6 | 1.16.3 | 1.14.5 | 1.17.2 |
| numpy | 1.13.3 | 1.13.3 | 1.13.3 | 1.14.5 | 1.14.6 | 1.16.3 | 1.14.5 | 1.17.4 |
+-------------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| onnx | N/A | N/A | N/A | 1.2.1 | 1.2.1 | 1.4.1 | 1.4.1 | 1.6.0 |
| onnx | N/A | N/A | N/A | 1.2.1 | 1.2.1 | 1.4.1 | 1.4.1 | 1.4.1 |
+-------------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+
| keras-mxnet | N/A | N/A | N/A | N/A | 2.2.2 | 2.2.4.1 | 2.2.4.1 | 2.2.4.2 |
| keras-mxnet | N/A | N/A | N/A | N/A | 2.2.2 | 2.2.4.1 | 2.2.4.1 | 2.2.4.1 |
+-------------------------+--------------+-------------+-------------+-------------+-------------+-------------+-------------+-------------+

The Docker images extend Ubuntu 16.04.
Expand Down
78 changes: 40 additions & 38 deletions src/sagemaker/pytorch/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ SageMaker PyTorch Estimators and Models

With PyTorch Estimators and Models, you can train and host PyTorch models on Amazon SageMaker.

Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``.
Supported versions of PyTorch: ``0.4.0``, ``1.0.0``, ``1.1.0``, ``1.2.0``, ``1.3.1``, ``1.4.0``.

We recommend that you use the latest supported version, because that's where we focus most of our development efforts.

Expand Down Expand Up @@ -37,43 +37,45 @@ This Python version applies to both the Training Job, created by fit, and the En

The PyTorch Docker images have the following dependencies installed:

+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| Dependencies | pytorch 0.4.0 | pytorch 1.0.0 | pytorch 1.1.0 | pytorch 1.2.0 | pytorch 1.3.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| boto3 | >=1.7.35 | >=1.9.11 | 1.9.82 | 1.9.249 | 1.10.34 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| botocore | >=1.10.35 | >=1.12.11 | >= 1.12.11 | 1.12.249 | 1.13.34 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| CUDA (GPU image only) | 9.0 | 9.0 | 10.1 | 10.0 | 10.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| numpy | >=1.14.3 | >=1.15.2 | 1.16.4 | 1.16.4 | 1.16.4 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| Pillow | >=5.1.0 | >=5.2.0 | 6.0.0 | 5.4.1 | 6.2.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| pip | >=10.0.1 | >=18.0 | >=18.0 | 19.3 | 19.3.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| python-dateutil | >=2.7.3 | >=2.7.3 | >=2.7.3 | 2.8.0 | 2.8.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| retrying | >=1.3.3 | >=1.3.3 | 1.3.3 | 1.3.3 | 1.3.3 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| s3transfer | >=0.1.13 | >=0.1.13 | >=0.1.13 | 0.2.1 | 0.2.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| sagemaker-containers | >=2.1.0 | >=2.1.0 | 2.4.10.post0 | 2.5.4 | 2.5.4 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| sagemaker-inference (inference only) | N/A | N/A | N/A | N/A | 1.1.2 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| sagemaker-pytorch-container | 1.0 | 1.1 | 1.2 | 1.2 | 1.3 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| setuptools | >=39.2.0 | >=40.4.3 | >=40.4.3 | 41.4.0 | 42.0.2.post20191203 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| six | >=1.11.0 | >=1.11.0 | 1.12.0 | 1.12.0 | 1.12.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| torch | 0.4.0 | 1.0.0 | 1.1.0 | 1.2.0 | 1.3.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| torchvision | 0.2.1 | 0.2.1 | 0.3.0 | 0.4.0a0+9232c4a | 0.4.2 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
| Python | 2.7 or 3.5 | 2.7 or 3.6 | 2.7 or 3.6 | 2.7 or 3.6 | 2.7 or 3.6 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| Dependencies | pytorch 0.4.0 | pytorch 1.0.0 | pytorch 1.1.0 | pytorch 1.2.0 | pytorch 1.3.1 | pytorch 1.4.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| boto3 | >=1.7.35 | >=1.9.11 | 1.9.82 | 1.9.249 | 1.10.34 | 1.12.4 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| botocore | >=1.10.35 | >=1.12.11 | >= 1.12.11 | 1.12.249 | 1.13.34 | 1.15.4 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| CUDA (GPU image only) | 9.0 | 9.0 | 10.1 | 10.0 | 10.1 | 10.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| numpy | >=1.14.3 | >=1.15.2 | 1.16.4 | 1.16.4 | 1.16.4 | 1.16.4 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| Pillow | >=5.1.0 | >=5.2.0 | 6.0.0 | 5.4.1 | 6.2.1 | 6.2.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| pip | >=10.0.1 | >=18.0 | >=18.0 | 19.3 | 19.3.1 | 20.0.2 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| python-dateutil | >=2.7.3 | >=2.7.3 | >=2.7.3 | 2.8.0 | 2.8.0 | 2.8.1 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| retrying | >=1.3.3 | >=1.3.3 | 1.3.3 | 1.3.3 | 1.3.3 | 1.3.3 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| s3transfer | >=0.1.13 | >=0.1.13 | >=0.1.13 | 0.2.1 | 0.2.1 | 0.3.3 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| sagemaker-containers | >=2.1.0 | >=2.1.0 | 2.4.10.post0 | 2.5.4 | 2.5.4 | 2.5.4 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| sagemaker-inference (inference only) | N/A | N/A | N/A | N/A | 1.1.2 | 1.1.2 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| sagemaker-pytorch-container | 1.0 | 1.1 | 1.2 | 1.2 | 1.3 | N/A |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| sagemaker-pytorch-inference | N/A | N/A | N/A | N/A | N/A | 1.1.2 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| setuptools | >=39.2.0 | >=40.4.3 | >=40.4.3 | 41.4.0 | 42.0.2.post20191203 | N/A |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| six | >=1.11.0 | >=1.11.0 | 1.12.0 | 1.12.0 | 1.12.0 | 1.12.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| torch | 0.4.0 | 1.0.0 | 1.1.0 | 1.2.0 | 1.3.1 | 1.4.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| torchvision | 0.2.1 | 0.2.1 | 0.3.0 | 0.4.0a0+9232c4a | 0.4.2 | 0.5.0 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+
| Python | 2.7 or 3.5 | 2.7 or 3.6 | 2.7 or 3.6 | 2.7 or 3.6 | 2.7 or 3.6 | 2.7 or 3.6 |
+---------------------------------------+---------------+----------------+---------------+-----------------+---------------------+---------------------+

The Docker images extend Ubuntu 16.04.

Expand Down
2 changes: 1 addition & 1 deletion src/sagemaker/pytorch/defaults.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
The default version is no longer updated so as to not break existing workflows.
"""

LATEST_VERSION = "1.3.1"
LATEST_VERSION = "1.4.0"
"""The latest version of PyTorch included in the SageMaker pre-built Docker images."""

PYTHON_VERSION = "py3"
Expand Down
26 changes: 13 additions & 13 deletions src/sagemaker/tensorflow/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,41 +30,41 @@ SageMaker TensorFlow Docker containers
The latest containers include the following Python packages:

+--------------------------------+---------------+---------------+
| Dependencies | TF 1.15 | TF 2.0 |
| Dependencies | TF 1.15.2 | TF 2.1 |
+--------------------------------+---------------+---------------+
| awscli | 1.16.296 | 1.16.303 |
| awscli | 1.18.1 | 1.18.3 |
+--------------------------------+---------------+---------------+
| boto3 | Latest | Latest |
| boto3 | 1.12.1 | 1.12.3 |
+--------------------------------+---------------+---------------+
| botocore | Latest | Latest |
| botocore | 1.15.1 | 1.15.3 |
+--------------------------------+---------------+---------------+
| h5py | 2.9.0 | 2.10.0 |
| h5py | 2.10.0 | 2.10.0 |
+--------------------------------+---------------+---------------+
| horovod | 0.18.2 | 0.18.2 |
+--------------------------------+---------------+---------------+
| keras | 2.3.1 | 2.3.1 |
+--------------------------------+---------------+---------------+
| mpi4py | 3.0.2 | 3.0.3 |
+--------------------------------+---------------+---------------+
| numpy | 1.17.4 | 1.17.4 |
| numpy | 1.18.1 | 1.18.1 |
+--------------------------------+---------------+---------------+
| pandas | 0.24.2 | 0.25.3 |
| pandas | 0.24.2 | 1.0.1 |
+--------------------------------+---------------+---------------+
| pip | 19.3.1 | 19.3.1 |
| pip | 20.0.2 | 20.0.2 |
+--------------------------------+---------------+---------------+
| Pillow | 6.1.0 | 6.2.1 |
| Pillow | 6.2.1 | 7.0.0 |
+--------------------------------+---------------+---------------+
| Python | 2.7 or 3.6 | 2.7 or 3.6 |
+--------------------------------+---------------+---------------+
| requests | 2.22.0 | 2.22.0 |
+--------------------------------+---------------+---------------+
| sagemaker-containers | >=2.6.1 | >=2.6.2 |
| sagemaker-containers | 2.7.0 | 2.8.0 |
+--------------------------------+---------------+---------------+
| sagemaker-tensorflow-container | 1.15.0.1.0.0 | 2.0.0.1.0.0 |
| sagemaker-tensorflow-container | 1.15.0.1.1.0 | 2.0.0.1.1.0 |
+--------------------------------+---------------+---------------+
| scipy | 1.2.2 | 1.2.2 |
| scipy | 1.2.2 | 1.4.1 |
+--------------------------------+---------------+---------------+
| tensorflow | 1.15.0 | 2.0.0 |
| tensorflow | 1.15.2 | 2.1.0 |
+--------------------------------+---------------+---------------+

Script Mode TensorFlow Docker images support both Python 2.7 and Python 3.6. The Docker images extend Ubuntu 16.04.
Expand Down
4 changes: 2 additions & 2 deletions src/sagemaker/tensorflow/defaults.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
This is no longer updated so as to not break existing workflows.
"""

LATEST_VERSION = "2.0.0"
LATEST_VERSION = "2.1.0"
"""The latest version of TensorFlow included in the SageMaker pre-built Docker images."""

LATEST_PY2_VERSION = "2.0.0"
LATEST_PY2_VERSION = "2.1.0"
4 changes: 2 additions & 2 deletions src/sagemaker/tensorflow/estimator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,12 +199,12 @@ class TensorFlow(Framework):

LATEST_VERSION = defaults.LATEST_VERSION

_LATEST_1X_VERSION = "1.15.0"
_LATEST_1X_VERSION = "1.15.2"

_LOWEST_SCRIPT_MODE_ONLY_VERSION = [1, 13]
# 2.0.0 still supports py2
# we will need to update this version number if future versions still support py2
_HIGHEST_PYTHON_2_VERSION = [2, 0, 0]
_HIGHEST_PYTHON_2_VERSION = [2, 1, 0]

def __init__(
self,
Expand Down
13 changes: 13 additions & 0 deletions tests/integ/test_pytorch_train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

from sagemaker.pytorch.estimator import PyTorch
from sagemaker.pytorch.model import PyTorchModel
from sagemaker.pytorch.defaults import LATEST_PY2_VERSION
from sagemaker.utils import sagemaker_timestamp

MNIST_DIR = os.path.join(DATA_DIR, "pytorch_mnist")
Expand All @@ -38,6 +39,10 @@ def fixture_training_job(sagemaker_session, pytorch_full_version, cpu_instance_t

@pytest.mark.canary_quick
@pytest.mark.regional_testing
@pytest.mark.skipif(
PYTHON_VERSION == "py2",
reason="Python 2 is supported by PyTorch {} and lower versions.".format(LATEST_PY2_VERSION),
)
def test_sync_fit_deploy(pytorch_training_job, sagemaker_session, cpu_instance_type):
# TODO: add tests against local mode when it's ready to be used
endpoint_name = "test-pytorch-sync-fit-attach-deploy{}".format(sagemaker_timestamp())
Expand All @@ -55,6 +60,10 @@ def test_sync_fit_deploy(pytorch_training_job, sagemaker_session, cpu_instance_t


@pytest.mark.local_mode
@pytest.mark.skipif(
PYTHON_VERSION == "py2",
reason="Python 2 is supported by PyTorch {} and lower versions.".format(LATEST_PY2_VERSION),
)
def test_fit_deploy(sagemaker_local_session, pytorch_full_version):
pytorch = PyTorch(
entry_point=MNIST_SCRIPT,
Expand All @@ -79,6 +88,10 @@ def test_fit_deploy(sagemaker_local_session, pytorch_full_version):
predictor.delete_endpoint()


@pytest.mark.skipif(
PYTHON_VERSION == "py2",
reason="Python 2 is supported by PyTorch {} and lower versions.".format(LATEST_PY2_VERSION),
)
def test_deploy_model(pytorch_training_job, sagemaker_session, cpu_instance_type):
endpoint_name = "test-pytorch-deploy-model-{}".format(sagemaker_timestamp())

Expand Down
Loading