fix: support estimator output path parameterization

.githooks/pre-push

@@ -12,5 +12,5 @@ start_time=`date +%s`
 tox -e sphinx,doc8 --parallel all
 ./ci-scripts/displaytime.sh 'sphinx,doc8' $start_time
 start_time=`date +%s`
-tox -e py36,py37,py38 --parallel all -- tests/unit
-./ci-scripts/displaytime.sh 'py36,py37,py38 unit' $start_time
+tox -e py36,py37,py38,py39 --parallel all -- tests/unit
+./ci-scripts/displaytime.sh 'py36,py37,py38,py39 unit' $start_time

.readthedocs.yml → .readthedocs.yaml

@@ -4,13 +4,19 @@
 
 version: 2
 
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.9"
+
+
 python:
-  version: 3.6
   install:
     - method: pip
       path: .
     - requirements: doc/requirements.txt
 
+
 sphinx:
   configuration: doc/conf.py
   fail_on_warning: true # http://www.sphinx-doc.org/en/master/man/sphinx-build.html#id6

CHANGELOG.md

@@ -1,5 +1,223 @@
 # Changelog
 
+## v2.89.0 (2022-05-11)
+
+### Features
+
+ * Add PT 1.11 support
+ * add validation specification
+
+### Bug Fixes and Other Changes
+
+ * repack model locally when local_code local mode
+
+### Documentation Changes
+
+ * smdmp 1.8.1 release note
+
+## v2.88.3 (2022-05-06)
+
+### Bug Fixes and Other Changes
+
+ * deprecate: Remove deprecated argument s3_data_distribution_type
+ * Feat/jumpstart model table update
+
+## v2.88.2 (2022-05-02)
+
+### Bug Fixes and Other Changes
+
+ * Automl integ describe job check
+ * Implement subclass compatibility for workflow pipeline job steps
+
+## v2.88.1 (2022-04-27)
+
+### Bug Fixes and Other Changes
+
+ * Add encryption setting to tar_and_upload_dir method
+
+## v2.88.0 (2022-04-26)
+
+### Features
+
+ * jumpstart notebook utils -- list model ids, scripts, tasks, frameworks
+
+### Bug Fixes and Other Changes
+
+ * local mode printing of credentials during docker login closes #2180
+ * disable endpoint context test
+
+### Documentation Changes
+
+ * sm model parallel 1.8.0 release notes
+
+## v2.87.0 (2022-04-20)
+
+### Features
+
+ * Add Jumpstart example notebooks
+ * add Tensorflow and Pytorch version for SM Training Compiler and expand to regular regions
+
+### Bug Fixes and Other Changes
+
+ * integs for training compiler in non-PDX regions
+ * TrainingStep cache misses due to timestamp based job name
+ * retry context delete
+ * Add more logging when unexpected number of artifacts found
+
+## v2.86.2 (2022-04-14)
+
+### Bug Fixes and Other Changes
+
+ * #using uuid to randomize, otherwise system timestamp is used
+
+## v2.86.1 (2022-04-13)
+
+### Bug Fixes and Other Changes
+
+ * xgboost, sklearn network isolation for jumpstart
+
+### Documentation Changes
+
+ * fix minor typo
+
+## v2.86.0 (2022-04-12)
+
+### Features
+
+ * Adds Spark Processing Notebook to Notebook Tests
+
+## v2.85.0 (2022-04-11)
+
+### Features
+
+ * update lambda code on pipeline create/update/upsert for Lamb…
+ * jumpstart model url
+ * add serverless inference image_uri retrieve support
+
+### Bug Fixes and Other Changes
+
+ * Add back the Fix for Pipeline variables related customer issues
+ * Support file URIs in ProcessingStep's code parameter
+
+## v2.84.0 (2022-04-07)
+
+### Features
+
+ * dependabot integ - move all deps to requirements.txt
+ * add xgboost framework version 1.5-1
+
+## v2.83.0 (2022-04-04)
+
+### Features
+
+ * Hugging Face Transformers 4.17 for TF 2.6
+
+### Bug Fixes and Other Changes
+
+ * IOC image version select issue
+
+## v2.82.2 (2022-04-01)
+
+### Bug Fixes and Other Changes
+
+ * Revert "fix: Fix Pipeline variables related customer issues (#2959)"
+ * Refactor repack_model script injection, fixes tar.gz error
+
+## v2.82.1 (2022-03-31)
+
+### Bug Fixes and Other Changes
+
+ * Update Inferentia Image URI Config
+ * Fix Pipeline variables related customer issues
+ * more logging info for static pipeline test data setup
+
+## v2.82.0 (2022-03-30)
+
+### Features
+
+ * pluggable instance fallback mechanism, add CapacityError
+ * support passing Env Vars to local mode training
+
+## v2.81.1 (2022-03-29)
+
+### Bug Fixes and Other Changes
+
+ * Update black-check version, add support for Spark 3.1 Processing
+
+## v2.81.0 (2022-03-26)
+
+### Features
+
+ * Retrieve data configuration
+ * enable EnableInterContainerTrafficEncryption for model monitoring
+ * Hugging Face Transformers 4.17 for PT 1.10
+
+### Bug Fixes and Other Changes
+
+ * remove `new` from serverless
+ * temporarily skip tests impacted by data inconsistency
+ * Implement override solution for pipeline variables
+
+### Documentation Changes
+
+ * add documentation for image_uri serverless use case
+ * minor fixes for smddp 1.4.0 doc
+
+## v2.80.0 (2022-03-18)
+
+### Features
+
+ * Add support for TF2.7
+ * Add support for TF 2.8
+ * TF242 ioc support
+ * Add support for TF 2.6.3
+ * Support for remote docker host
+ * AutoGluon 0.3.2 and 0.4.0 image_uris
+
+### Bug Fixes and Other Changes
+
+ * Align max_wait definitions in EstimaorBase and Estimator
+ * Add JumpStart model table build notification
+ * gpu integs CapacityError - fallback to available compute
+ * gpu integs CapacityError - fallback to available compute
+ * jumpstart docs network isolation
+
+### Documentation Changes
+
+ * sagemaker distributed model parallel 1.7.0 doc
+
+## v2.79.0 (2022-03-16)
+
+### Features
+
+ * Inferentia Neuron support for HuggingFace
+ * custom base job name for jumpstart models/estimators
+ * Python 3.9 for readthedocs
+
+### Bug Fixes and Other Changes
+
+ * container env generation for S3 URI and add test for the same
+
+### Documentation Changes
+
+ * the SageMaker distributed data parallel v1.4.0 release
+ * update sagemaker training compiler docstring
+ * smddp doc update
+
+## v2.78.0 (2022-03-07)
+
+### Features
+
+ * TensorFlow 2.4 for Neo
+ * Data Serializer
+
+### Bug Fixes and Other Changes
+
+ * Style update in DataSerializer
+ * Remove sagemaker_job_name from hyperparameters in TrainingStep
+ * reorganize test files for workflow
+ * update code to get commit_id in codepipeline
+
 ## v2.77.1 (2022-02-25)
 
 ### Bug Fixes and Other Changes
@@ -11,7 +229,7 @@
 ### Features
 
  * override jumpstart content bucket
- * jumpstart model id suggestions
+ * jumpstart model ID suggestions
  * adding customer metadata support to registermodel step
 
 ### Bug Fixes and Other Changes

MANIFEST.in

@@ -1,6 +1,7 @@
 recursive-include src/sagemaker *.py
 
 include src/sagemaker/image_uri_config/*.json
+recursive-include requirements *
 
 include VERSION
 include LICENSE.txt

VERSION

@@ -1 +1 @@
-2.77.2.dev0
+2.89.1.dev0

doc/_static/js/datatable.js

@@ -0,0 +1,4 @@
+$(document).ready( function () {
+    $('table.datatable').DataTable();
+    $('a.external').attr('target', '_blank');
+} );

doc/api/training/distributed.rst

@@ -4,8 +4,28 @@ SageMaker distributed training libraries offer both data parallel and model para
 They combine software and hardware technologies to improve inter-GPU and inter-node communications.
 They extend SageMaker’s training capabilities with built-in options that require only small code changes to your training scripts.
 
+.. _sdp_api_docs_toc:
+
+The SageMaker Distributed Data Parallel Library
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. toctree::
+    :maxdepth: 2
+
+    smd_data_parallel
+    sdp_versions/latest
+    smd_data_parallel_use_sm_pysdk
+    smd_data_parallel_release_notes/smd_data_parallel_change_log
+
+.. _smp_api_docs_toc:
+
+The SageMaker Distributed Model Parallel Library
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 .. toctree::
-   :maxdepth: 3
+   :maxdepth: 2
 
-   smd_data_parallel
    smd_model_parallel
+   smp_versions/latest
+   smd_model_parallel_general
+   smd_model_parallel_release_notes/smd_model_parallel_change_log

doc/api/training/sdp_versions/archives.rst

@@ -0,0 +1,8 @@
+.. _smddp-version-archive:
+
+.. toctree::
+    :maxdepth: 1
+
+    v1_2_x.rst
+    v1_1_x.rst
+    v1_0_0.rst

doc/api/training/sdp_versions/latest.rst

@@ -1,9 +1,47 @@
+.. _sdp_api_docs:
 
-Version 1.2.x (Latest)
+#############################################
+Use the Library to Adapt Your Training Script
+#############################################
+
+This section contains the SageMaker distributed data parallel API documentation.
+If you are a new user of this library, it is recommended you use this guide alongside
+`SageMaker's Distributed Data Parallel Library
+<https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel.html>`_.
+
+The library provides framework-specific APIs for TensorFlow and PyTorch.
+
+Select the latest or one of the previous versions of the API documentation
+depending on the version of the library you use.
+
+.. important::
+
+   The distributed data parallel library supports training jobs using CUDA 11 or later.
+   When you define a :class:`sagemaker.tensorflow.estimator.TensorFlow` or
+   :class:`sagemaker.pytorch.estimator.PyTorch`
+   estimator with the data parallel library enabled,
+   SageMaker uses CUDA 11. When you extend or customize your own training image,
+   you must use a base image with CUDA 11 or later. See
+   `SageMaker Python SDK's distributed data parallel library APIs
+   <https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html#data-parallel-use-python-skd-api>`_
+   for more information.
+
+Version 1.4.0 (Latest)
 ======================
 
 .. toctree::
    :maxdepth: 1
 
-   latest/smd_data_parallel_pytorch.rst
-   latest/smd_data_parallel_tensorflow.rst
+   latest/smd_data_parallel_pytorch
+   latest/smd_data_parallel_tensorflow
+
+Documentation Archive
+=====================
+
+To find the API documentation for the previous versions of the library,
+choose one of the following:
+
+.. toctree::
+   :maxdepth: 1
+
+   archives