Merge branch 'master' into pandas_dtypes_lower_fix

Author: mufaddal-rohawala

.flake8

@@ -3,3 +3,4 @@ application_import_names = sagemaker, tests
 import-order-style = google
 per-file-ignores =
     tests/unit/test_tuner.py: F405
+    src/sagemaker/config/config_schema.py: E501

.gitignore

@@ -30,6 +30,7 @@ env/
 .vscode/
 **/tmp
 .python-version
+*.html
 **/_repack_script_launcher.sh
 tests/data/**/_repack_model.py
 tests/data/experiment/sagemaker-dev-1.0.tar.gz

CHANGELOG.md

@@ -1,5 +1,179 @@
 # Changelog
 
+## v2.152.0 (2023-05-04)
+
+### Features
+
+ * add support for lineage visualization using pyvis
+ * Expose Experiment class publicly
+ * PyTorch 1.13 release
+
+### Bug Fixes and Other Changes
+
+ * Change data_type argument to dtype to keep consistent with D…
+ * Skip edge test
+ * make RemoteExecutor context manager non-blocking on pending futures
+ * Add inferentia2 DLC images for djl framework
+ * Fix typo in using_pytorch.rst
+ * Unable to attach estimator to training job when KeepAlivePeriodInSeconds specified
+ * update LMI container image
+ * Update Clarify SHAPConfig baseline to allow JSON structures
+
+### Documentation Changes
+
+ * Fix broken link in DJL SageMaker docs
+ * currency update for the SageMaker data parallelism lib
+ * SM model parallel library v1.15.0 release note
+
+## v2.151.0 (2023-04-27)
+
+### Features
+
+ * Update Transformers 4.26 - TensorFlow 2.11.0 Image URI
+ * Add Extra Parameters to Lambda Function Wrapper
+
+### Bug Fixes and Other Changes
+
+ * Add kms key support for Model registration
+ * Enable inference recommender slow tests
+ * Pass sagemaker session to downstream s3 calls
+ * Add ap-south-1 to no p3 regions
+ * skip test for p2 instance for TF2.12 and above
+
+### Documentation Changes
+
+ * Fix minor misses from the remote function doc release
+
+## v2.150.0 (2023-04-26)
+
+### Features
+
+ * Introduce TensorBoard app class
+
+### Bug Fixes and Other Changes
+
+ * Update data wrangler images
+
+## v2.149.0 (2023-04-25)
+
+### Features
+
+ * Support TF2.12 SageMaker DLC
+
+### Bug Fixes and Other Changes
+
+ * update the doc for Join function
+ * change s3UploadMode of sagemaker clarify processing output for computer vision jobs.
+
+### Documentation Changes
+
+ * Add Remote Function updates
+
+## v2.148.0 (2023-04-20)
+
+### Features
+
+ * [huggingface] Add `torch.distributed` support for Trainium and `torchrun`
+ * Add PyTorch 2.0 to SDK
+
+### Bug Fixes and Other Changes
+
+ * updating batch transform job in monitoring schedule
+
+## v2.147.0 (2023-04-18)
+
+### Features
+
+ * support different types of deletion mode
+
+## v2.146.1 (2023-04-17)
+
+### Bug Fixes and Other Changes
+
+ * skip failing tests temporarily
+ * Added ml.p4d and ml.p4de as supported instances for DeepSpeed
+
+### Documentation Changes
+
+ * Add Model Registry Model Collection
+
+## v2.146.0 (2023-04-13)
+
+### Features
+
+ * Add support for JSON model inputs for Clarify Processor
+
+### Bug Fixes and Other Changes
+
+ * Feature/list collection
+ * improve reliability of Run integration test
+ * Add a comment that smdataparallel lib excludes tf 2.12 support
+
+### Documentation Changes
+
+ * Update reference to load run method in documentation
+
+## v2.145.0 (2023-04-06)
+
+### Features
+
+ * add support for async inline error notifications
+ * Add methods for feature group to list feature metadata parameters and tags
+ * Support huggingface hub model_id for DJL Models
+
+### Bug Fixes and Other Changes
+
+ * load_sagemaker_config should lazy initialize a default S3 resource
+
+## v2.144.0 (2023-04-05)
+
+### Features
+
+ * support create Clarify explainer enabled endpoint for Clarify Online Explainability
+ * Combined inference and training script artifact
+ * jumpstart instance types
+ * Deprecation warning for framework profiling for TF 2.12 and on, PT 2.0 and on
+
+### Bug Fixes and Other Changes
+
+ * always delete temporary directory even during exception
+ * Fixes the completion_criteria_config dict in the to_input_req method
+ * Update CHANGELOG.md
+
+### Documentation Changes
+
+ * Update SageMaker Debugger doc
+
+## v2.143.0 (2023-03-29)
+
+### Features
+
+ * Support for SageMaker SDK Defaults
+
+### Bug Fixes and Other Changes
+
+ * update feature store offline s3 path used in tests
+
+## v2.142.0 (2023-03-27)
+
+### Features
+
+ * combined model + script artifact
+
+## v2.141.0 (2023-03-24)
+
+### Features
+
+ * AutoGluon 0.7.0 image_uris update
+ * Add DJL FasterTransformer image uris
+ * EMR step runtime role support
+ * locations for EMR configuration and Spark dependencies
+ * Adding support for 1P Algorithms in ZAZ, ZRH, HYD, MEL
+
+### Documentation Changes
+
+ * Update FeatureGroup kms key id documentation
+
 ## v2.140.1 (2023-03-21)
 
 ### Bug Fixes and Other Changes

README.rst

@@ -133,6 +133,8 @@ To run the integration tests, the following prerequisites must be met
 1. AWS account credentials are available in the environment for the boto3 client to use.
 2. The AWS account has an IAM role named :code:`SageMakerRole`.
    It should have the AmazonSageMakerFullAccess policy attached as well as a policy with `the necessary permissions to use Elastic Inference <https://docs.aws.amazon.com/sagemaker/latest/dg/ei-setup.html>`__.
+3. To run remote_function tests, dummy ecr repo should be created. It can be created by running -
+    :code:`aws ecr create-repository --repository-name remote-function-dummy-container`
 
 We recommend selectively running just those integration tests you'd like to run. You can filter by individual test function names with:
 

VERSION

@@ -1 +1 @@
-2.140.2.dev0
+2.152.1.dev0

doc/amazon_sagemaker_debugger.rst

@@ -4,6 +4,12 @@ Amazon SageMaker Debugger
 #########################
 
 
+.. warning::
+
+  This page is no longer supported for maintenence. The live documentation is at `Debug and Profile Training Jobs Using Amazon SageMaker Debugger <https://docs.aws.amazon.com/sagemaker/latest/dg/train-debugger.html>`_
+  and `Debugger API <https://sagemaker.readthedocs.io/en/stable/api/training/debugger.html>`_.
+
+
 Amazon SageMaker Debugger allows you to detect anomalies while training your machine learning model by emitting relevant data during training, storing the data and then analyzing it.
 
 .. contents::

doc/api/inference/explainer.rst

@@ -0,0 +1,16 @@
+Online Explainability
+---------------------
+
+This module contains classes related to Amazon Sagemaker Clarify Online Explainability
+
+.. automodule:: sagemaker.explainer.explainer_config
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: sagemaker.explainer.clarify_explainer_config
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+

doc/api/inference/model_collection.rst

@@ -0,0 +1,8 @@
+Model Collection
+----------------
+
+.. automodule:: sagemaker.collection
+    :members:
+    :undoc-members:
+    :show-inheritance:
+

doc/api/prep_data/feature_store.rst

@@ -1,7 +1,7 @@
 Feature Store APIs
 ------------------
 
-Feature group
+Feature Group
 *************
 
 .. autoclass:: sagemaker.feature_store.feature_group.FeatureGroup
@@ -18,7 +18,7 @@ Feature group
     :show-inheritance:
 
 
-Feature definition
+Feature Definition
 ******************
 
 .. autoclass:: sagemaker.feature_store.feature_definition.FeatureDefinition
@@ -77,10 +77,46 @@ Inputs
     :members:
     :show-inheritance:
 
+.. autoclass:: sagemaker.feature_store.inputs.ResourceEnum
+    :members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.SearchOperatorEnum
+    :members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.SortOrderEnum
+    :members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.FilterOperatorEnum
+    :members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.Filter
+    :members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.Identifier
+    :members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.feature_store.inputs.FeatureParameter
+    :members:
+    :show-inheritance:
+
 
 Dataset Builder
 ***************
 
 .. autoclass:: sagemaker.feature_store.dataset_builder.DatasetBuilder
     :members:
     :show-inheritance:
+
+
+Feature Store
+*************
+
+.. autoclass:: sagemaker.feature_store.feature_store.FeatureStore
+    :members:
+    :show-inheritance:

doc/api/training/debugger.rst

@@ -10,8 +10,13 @@ Configure the Debugger-specific parameters when constructing
 a SageMaker estimator to gain visibility and insights
 into your training job.
 
+.. contents::
+
 .. currentmodule:: sagemaker.debugger
 
+Debugger Rule APIs
+~~~~~~~~~~~~~~~~~~
+
 .. autoclass:: get_rule_container_image_uri
     :show-inheritance:
 
@@ -44,6 +49,9 @@ into your training job.
     :show-inheritance:
     :inherited-members:
 
+Debugger Configuration APIs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 .. autoclass:: CollectionConfig
     :show-inheritance:
 
@@ -56,6 +64,21 @@ into your training job.
 .. autoclass:: ProfilerConfig
     :show-inheritance:
 
+Debugger Configuration APIs for Framework Profiling (Deprecated)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. warning::
+
+  SageMaker Debugger deprecates the framework profiling feature starting from TensorFlow 2.11 and PyTorch 2.0. You can still use the feature in the previous versions of the frameworks and SDKs as follows.
+
+  * SageMaker Python SDK <= v2.130.0
+  * PyTorch >= v1.6.0, < v2.0
+  * TensorFlow >= v2.3.1, < v2.11
+
+  With the deprecation, SageMaker Debugger discontinues support for the APIs below this note.
+
+  See also `Amazon SageMaker Debugger Release Notes: March 16, 2023 <https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-release-notes.html#debugger-release-notes-20230315>`_.
+
 .. autoclass:: FrameworkProfile
     :show-inheritance:
 

doc/api/training/sdp_versions/latest.rst

@@ -26,7 +26,7 @@ depending on the version of the library you use.
    <https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html#data-parallel-use-python-skd-api>`_
    for more information.
 
-For versions between 1.4.0 and 1.7.0 (Latest)
+For versions between 1.4.0 and 1.8.0 (Latest)
 =============================================
 
 .. toctree::