Merge branch 'master' into master

Author: szamarin

.gitignore

@@ -30,5 +30,6 @@ env/
 .vscode/
 **/tmp
 .python-version
-**/_repack_model.py
-**/_repack_script_launcher.sh
+**/_repack_script_launcher.sh
+tests/data/**/_repack_model.py
+tests/data/experiment/sagemaker-dev-1.0.tar.gz

CHANGELOG.md

@@ -1,5 +1,119 @@
 # Changelog
 
+## v2.125.0 (2022-12-19)
+
+### Features
+
+ * add RandomSeed to support reproducible HPO
+
+### Bug Fixes and Other Changes
+
+ * Correct SageMaker Clarify API docstrings by changing JSONPath to JMESPath
+
+## v2.124.0 (2022-12-16)
+
+### Features
+
+ * Doc update for TableFormatEnum
+ * Add p4de to smddp supported instance types
+ * Add disable_profiler field in config and propagate changes
+ * Added doc update for dataset builder
+
+### Bug Fixes and Other Changes
+
+ * Use Async Inference Config when available for endpoint update
+
+### Documentation Changes
+
+ * smdistributed libraries release notes
+
+## v2.123.0 (2022-12-15)
+
+### Features
+
+ * Add support for TF2.9.2 training images
+ * Add SageMaker Experiment
+
+## v2.122.0 (2022-12-14)
+
+### Features
+
+ * Feature Store dataset builder, delete_record, get_record, list_feature_group
+ * Add OSU region to frameworks for DLC
+
+### Bug Fixes and Other Changes
+
+ * the Hyperband support fix for the HPO
+ * unpin packaging version
+ * Remove content type image/jpg from analysis configuration schema
+
+## v2.121.2 (2022-12-12)
+
+### Bug Fixes and Other Changes
+
+ * Update for Tensorflow Serving 2.11 inference DLCs
+ * Revert "fix: type hint of PySparkProcessor __init__"
+ * Skip Bad Transform Test
+
+## v2.121.1 (2022-12-09)
+
+### Bug Fixes and Other Changes
+
+ * Pop out ModelPackageName from pipeline definition
+ * Fix failing jumpstart cache unit tests
+
+## v2.121.0 (2022-12-08)
+
+### Features
+
+ * Algorithms Region Expansion OSU/DXB
+
+### Bug Fixes and Other Changes
+
+ * FrameworkProcessor S3 uploads
+ * Add constraints file for apache-airflow
+
+## v2.120.0 (2022-12-07)
+
+### Features
+
+ * Add Neo image uri config for Pytorch 1.12
+ * Adding support for SageMaker Training Compiler in PyTorch estimator starting 1.12
+ * Update registries with new region account number mappings.
+ * Add DXB region to frameworks by DLC
+
+### Bug Fixes and Other Changes
+
+ * support idempotency for framework and spark processors
+
+## v2.119.0 (2022-12-03)
+
+### Features
+
+ * Add Code Owners file
+ * Added transform with monitoring pipeline step in transformer
+ * Update TF 2.9 and TF 2.10 inference DLCs
+ * make estimator accept json file as modelparallel config
+ * SageMaker Training Compiler does not support p4de instances
+ * Add support for SparkML v3.3
+
+### Bug Fixes and Other Changes
+
+ * Fix bug forcing uploaded tar to be named sourcedir
+ * Update local_requirements.txt PyYAML version
+ * refactoring : using with statement
+ * Allow Py 3.7 for MMS Test Docker env
+ * fix PySparkProcessor __init__ params type
+ * type hint of PySparkProcessor __init__
+ * Return ARM XGB/SKLearn tags if `image_scope` is `inference_graviton`
+ * Update scipy to 1.7.3 to support M1 development envs
+ * Fixing type hints for Spark processor that has instance type/count params in reverse order
+ * Add DeepAR ap-northeast-3 repository.
+ * Fix AsyncInferenceConfig documentation typo
+ * fix ml_inf to ml_inf1 in Neo multi-version support
+ * Fix type annotations
+ * add neo mvp region accounts
+
 ## v2.118.0 (2022-12-01)
 
 ### Features

VERSION

@@ -1 +1 @@
-2.118.1.dev0
+2.125.1.dev0

doc/amazon_sagemaker_model_building_pipeline.rst

@@ -453,7 +453,7 @@ Example:
             str_outputParam, int_outputParam, bool_outputParam, float_outputParam
        ],
     )
-    output_ref = step_lambda.OutputParameters["output1"]
+    output_ref = step_lambda.properties.Outputs["output1"]
 
 Where the lambda function with :code:`arn arn:aws:lambda:us-west-2:123456789012:function:sagemaker_test_lambda`
 should output like this:
@@ -479,7 +479,7 @@ Note that the output parameters can not be nested. Otherwise, the value will be
         }
     }
 
-This will be resolved as :code:`{"output1": "{\"nested_output1\":\"my-output\"}"}` by which if you refer :code:`step_lambda.OutputParameters["output1"]["nested_output1"]` later, a non-retryable client error will be thrown.
+This will be resolved as :code:`{"output1": "{\"nested_output1\":\"my-output\"}"}` by which if you refer :code:`step_lambda.properties.Outputs["output1"]["nested_output1"]` later, a non-retryable client error will be thrown.
 
 CallbackStep
 `````````````
@@ -503,7 +503,7 @@ Example:
         inputs={"arg1": "foo", "arg2": 5, "arg3": param},
         outputs=[outputParam],
     )
-    output_ref = step_callback.OutputParameters["output1]
+    output_ref = step_callback.properties.Outputs["output1]
 
 The output parameters cannot be nested. If the values are nested, they will be treated as a single string value. For example, a nested output value of
 
@@ -515,7 +515,7 @@ The output parameters cannot be nested. If the values are nested, they will be t
         }
     }
 
-is resolved as :code:`{"output1": "{\"nested_output1\":\"my-output\"}"}`. If you try to refer to :code:`step_callback.OutputParameters["output1"]["nested_output1"]` this will throw a non-retryable client error.
+is resolved as :code:`{"output1": "{\"nested_output1\":\"my-output\"}"}`. If you try to refer to :code:`step_callback.properties.Outputs["output1"]["nested_output1"]` this will throw a non-retryable client error.
 
 
 QualityCheckStep

doc/api/prep_data/feature_store.rst

@@ -72,3 +72,15 @@ Inputs
 .. autoclass:: sagemaker.feature_store.inputs.FeatureValue
     :members:
     :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.TableFormatEnum
+    :members:
+    :show-inheritance:
+
+
+Dataset Builder
+***************
+
+.. autoclass:: sagemaker.feature_store.dataset_builder.DatasetBuilder
+    :members:
+    :show-inheritance:

doc/api/training/sdp_versions/latest.rst

@@ -26,8 +26,8 @@ depending on the version of the library you use.
    <https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html#data-parallel-use-python-skd-api>`_
    for more information.
 
-Version 1.4.0, 1.4.1, 1.5.0 (Latest)
-====================================
+Version 1.4.0, 1.4.1, 1.5.0, 1.6.0 (Latest)
+===========================================
 
 .. toctree::
    :maxdepth: 1

doc/api/training/smd_data_parallel_release_notes/smd_data_parallel_change_log.rst

@@ -7,9 +7,51 @@ Release Notes
 New features, bug fixes, and improvements are regularly made to the SageMaker
 distributed data parallel library.
 
-SageMaker Distributed Data Parallel 1.5.0 Release Notes
+SageMaker Distributed Data Parallel 1.6.0 Release Notes
 =======================================================
 
+*Date: Dec. 15. 2022*
+
+**New Features**
+
+* New optimized SMDDP AllGather collective to complement the sharded data parallelism technique
+  in the SageMaker model parallelism library. For more information, see `Sharded data parallelism with SMDDP Collectives
+  <https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-extended-features-pytorch-sharded-data-parallelism.html#model-parallel-extended-features-pytorch-sharded-data-parallelism-smddp-collectives>`_
+  in the *Amazon SageMaker Developer Guide*.
+* Added support for Amazon EC2 ``ml.p4de.24xlarge`` instances. You can run data parallel training jobs
+  on ``ml.p4de.24xlarge`` instances with the SageMaker data parallelism library’s AllReduce collective.
+
+**Improvements**
+
+* General performance improvements of the SMDDP AllReduce collective communication operation.
+
+**Migration to AWS Deep Learning Containers**
+
+This version passed benchmark testing and is migrated to the following AWS Deep Learning Containers (DLC):
+
+- SageMaker training container for PyTorch v1.12.1
+
+  .. code::
+
+    763104351884.dkr.ecr.<region>.amazonaws.com/pytorch-training:1.12.1-gpu-py38-cu113-ubuntu20.04-sagemaker
+
+
+Binary file of this version of the library for `custom container
+<https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html#data-parallel-bring-your-own-container>`_ users:
+
+  .. code::
+
+    https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.12.1/cu113/2022-12-05/smdistributed_dataparallel-1.6.0-cp38-cp38-linux_x86_64.whl
+
+
+----
+
+Release History
+===============
+
+SageMaker Distributed Data Parallel 1.5.0 Release Notes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 *Date: Jul. 26. 2022*
 
 **Currency Updates**
@@ -38,12 +80,6 @@ Binary file of this version of the library for `custom container
 
     https://smdataparallel.s3.amazonaws.com/binary/pytorch/1.12.0/cu113/2022-07-01/smdistributed_dataparallel-1.5.0-cp38-cp38-linux_x86_64.whl
 
-
-----
-
-Release History
-===============
-
 SageMaker Distributed Data Parallel 1.4.1 Release Notes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/api/training/smd_model_parallel_release_notes/smd_model_parallel_change_log.rst

@@ -6,9 +6,60 @@ New features, bug fixes, and improvements are regularly made to the SageMaker
 distributed model parallel library.
 
 
-SageMaker Distributed Model Parallel 1.11.0 Release Notes
+SageMaker Distributed Model Parallel 1.13.0 Release Notes
 =========================================================
 
+*Date: Dec. 15. 2022*
+
+**New Features**
+
+* Sharded data parallelism now supports a new backend for collectives called *SMDDP Collectives*.
+  For supported scenarios, SMDDP Collectives are on by default for the AllGather operation.
+  For more information, see
+  `Sharded data parallelism with SMDDP Collectives
+  <https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-extended-features-pytorch-sharded-data-parallelism.html#model-parallel-extended-features-pytorch-sharded-data-parallelism-smddp-collectives>`_
+  in the *Amazon SageMaker Developer Guide*.
+* Introduced FlashAttention for DistributedTransformer to improve memory usage and computational
+  performance of models such as GPT2, GPTNeo, GPTJ, GPTNeoX, BERT, and RoBERTa.
+
+**Bug Fixes**
+
+* Fixed initialization of ``lm_head`` in DistributedTransformer to use a provided range
+  for initialization, when weights are not tied with the embeddings.
+
+**Improvements**
+
+* When a module has no parameters, we have introduced an optimization to execute
+  such a module on the same rank as its parent during pipeline parallelism.
+
+**Migration to AWS Deep Learning Containers**
+
+This version passed benchmark testing and is migrated to the following AWS Deep Learning Containers (DLC):
+
+- SageMaker training container for PyTorch v1.12.1
+
+  .. code::
+
+    763104351884.dkr.ecr.<region>.amazonaws.com/pytorch-training:1.12.1-gpu-py38-cu113-ubuntu20.04-sagemaker
+
+
+Binary file of this version of the library for `custom container
+<https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-sm-sdk.html#model-parallel-bring-your-own-container>`_ users:
+
+- For PyTorch 1.12.0
+
+  .. code::
+
+    https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.12.1/build-artifacts/2022-12-08-21-34/smdistributed_modelparallel-1.13.0-cp38-cp38-linux_x86_64.whl
+
+----
+
+Release History
+===============
+
+SageMaker Distributed Model Parallel 1.11.0 Release Notes
+---------------------------------------------------------
+
 *Date: August. 17. 2022*
 
 **New Features**
@@ -41,12 +92,7 @@ Binary file of this version of the library for `custom container
 
   .. code::
 
-    https://sagemaker-distributed-model-parallel.s3.us-west-2.amazonaws.com/pytorch-1.12.0/build-artifacts/2022-08-12-16-58/smdistributed_modelparallel-1.11.0-cp38-cp38-linux_x86_64.whl
-
-----
-
-Release History
-===============
+    https://sagemaker-distribu
 
 SageMaker Distributed Model Parallel 1.10.1 Release Notes
 ---------------------------------------------------------

doc/api/training/smp_versions/latest.rst

@@ -10,8 +10,8 @@ depending on which version of the library you need to use.
 To use the library, reference the
 **Common API** documentation alongside the framework specific API documentation.
 
-Version 1.11.0 (Latest)
-===========================================
+Version 1.11.0, 1.13.0 (Latest)
+===============================
 
 To use the library, reference the Common API documentation alongside the framework specific API documentation.
 

doc/experiments/index.rst

@@ -0,0 +1,10 @@
+############################
+Amazon SageMaker Experiments
+############################
+
+The SageMaker Python SDK supports to track and organize your machine learning workflow across SageMaker with jobs, such as Processing, Training and Transform, or locally.
+
+.. toctree::
+    :maxdepth: 2
+
+    sagemaker.experiments

doc/experiments/sagemaker.experiments.rst

@@ -0,0 +1,20 @@
+Experiments
+============
+
+Run
+-------------
+
+.. autoclass:: sagemaker.experiments.Run
+    :members:
+
+.. automethod:: sagemaker.experiments.load_run
+
+.. automethod:: sagemaker.experiments.list_runs
+
+.. autoclass:: sagemaker.experiments.SortByType
+    :members:
+    :undoc-members:
+
+.. autoclass:: sagemaker.experiments.SortOrderType
+    :members:
+    :undoc-members:

doc/index.rst

@@ -60,6 +60,16 @@ Orchestrate your SageMaker training and inference workflows with Airflow and Kub
     workflows/index
 
 
+****************************
+Amazon SageMaker Experiments
+****************************
+You can use Amazon SageMaker Experiments to track machine learning experiments.
+
+.. toctree::
+    :maxdepth: 2
+
+    experiments/index
+
 *************************
 Amazon SageMaker Debugger
 *************************