Merge branch 'master' into tf-2.10

Author: RadhikaB-97

CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## v2.114.0 (2022-10-26)
+
+### Features
+
+ * Graviton support for XGB and SKLearn frameworks
+ * Graviton support for PyTorch and Tensorflow frameworks
+ * do not expand estimator role when it is pipeline parameter
+ * added support for batch transform with model monitoring
+
+### Bug Fixes and Other Changes
+
+ * regex in tuning integs
+ * remove debugger environment var set up
+ * adjacent slash in s3 key
+ * Fix Repack step auto install behavior
+ * Add retry for airflow ParsingError
+
+### Documentation Changes
+
+ * doc fix
+
 ## v2.113.0 (2022-10-21)
 
 ### Features

VERSION

@@ -1 +1 @@
-2.113.1.dev0
+2.114.1.dev0

doc/frameworks/pytorch/using_pytorch.rst

@@ -293,107 +293,6 @@ using two ``ml.p4d.24xlarge`` instances:
 
     pt_estimator.fit("s3://bucket/path/to/training/data")
 
-.. _distributed-pytorch-training-on-trainium:
-
-Distributed PyTorch Training on Trainium
-========================================
-
-SageMaker Training on Trainium instances now supports the ``xla``
-package through ``torchrun``. With this, you do not need to manually pass RANK,
-WORLD_SIZE, MASTER_ADDR, and MASTER_PORT. You can launch the training job using the
-:class:`sagemaker.pytorch.estimator.PyTorch` estimator class
-with the ``torch_distributed`` option as the distribution strategy.
-
-.. note::
-
-  This ``torch_distributed`` support is available
-  in the SageMaker Trainium (trn1) PyTorch Deep Learning Containers starting v1.11.0.
-  To find a complete list of supported versions of PyTorch Neuron, see `Neuron Containers <https://github.com/aws/deep-learning-containers/blob/master/available_images.md#neuron-containers>`_ in the *AWS Deep Learning Containers GitHub repository*.
-
-  SageMaker Debugger and Profiler are currently not supported with Trainium instances.
-
-Adapt Your Training Script to Initialize with the XLA backend
--------------------------------------------------------------
-
-To initialize distributed training in your script, call
-`torch.distributed.init_process_group
-<https://pytorch.org/docs/master/distributed.html#torch.distributed.init_process_group>`_
-with the ``xla`` backend as shown below.
-
-.. code:: python
-
-    import torch.distributed as dist
-
-    dist.init_process_group('xla')
-
-SageMaker takes care of ``'MASTER_ADDR'`` and ``'MASTER_PORT'`` for you via ``torchrun``
-
-For detailed documentation about modifying your training script for Trainium, see `Multi-worker data-parallel MLP training using torchrun <https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/torch/torch-neuronx/tutorials/training/mlp.html?highlight=torchrun#multi-worker-data-parallel-mlp-training-using-torchrun>`_ in the *AWS Neuron Documentation*.
-
-**Currently Supported backends:**
-
--  ``xla`` for Trainium (Trn1) instances
-
-For up-to-date information on supported backends for Trainium instances, see `AWS Neuron Documentation <https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html>`_.
-
-Launching a Distributed Training Job on Trainium
-------------------------------------------------
-
-You can run multi-node distributed PyTorch training jobs on Trainium instances using the
-:class:`sagemaker.pytorch.estimator.PyTorch` estimator class.
-With ``instance_count=1``, the estimator submits a
-single-node training job to SageMaker; with ``instance_count`` greater
-than one, a multi-node training job is launched.
-
-With the ``torch_distributed`` option, the SageMaker PyTorch estimator runs a SageMaker
-training container for PyTorch Neuron, sets up the environment, and launches
-the training job using the ``torchrun`` command on each worker with the given information.
-
-**Examples**
-
-The following examples show how to run a PyTorch training using ``torch_distributed`` in SageMaker
-on one ``ml.trn1.2xlarge`` instance and two ``ml.trn1.32xlarge`` instances:
-
-.. code:: python
-
-    from sagemaker.pytorch import PyTorch
-
-    pt_estimator = PyTorch(
-        entry_point="train_ptddp.py",
-        role="SageMakerRole",
-        framework_version="1.11.0",
-        py_version="py38",
-        instance_count=1,
-        instance_type="ml.trn1.2xlarge",
-        distribution={
-            "torch_distributed": {
-                "enabled": True
-            }
-        }
-    )
-
-    pt_estimator.fit("s3://bucket/path/to/training/data")
-
-.. code:: python
-
-    from sagemaker.pytorch import PyTorch
-
-    pt_estimator = PyTorch(
-        entry_point="train_ptddp.py",
-        role="SageMakerRole",
-        framework_version="1.11.0",
-        py_version="py38",
-        instance_count=2,
-        instance_type="ml.trn1.32xlarge",
-        distribution={
-            "torch_distributed": {
-                "enabled": True
-            }
-        }
-    )
-
-    pt_estimator.fit("s3://bucket/path/to/training/data")
-
 *********************
 Deploy PyTorch Models
 *********************

src/sagemaker/estimator.py

@@ -865,10 +865,6 @@ def _prepare_debugger_for_training(self):
         self.debugger_rule_configs = self._prepare_debugger_rules()
         self._prepare_collection_configs()
         self._validate_and_set_debugger_configs()
-        if not self.debugger_hook_config:
-            if self.environment is None:
-                self.environment = {}
-            self.environment[DEBUGGER_FLAG] = "0"
 
     def _validate_and_set_debugger_configs(self):
         """Set defaults for debugging."""

src/sagemaker/fw_utils.py

@@ -146,6 +146,12 @@
 SMDISTRIBUTED_SUPPORTED_STRATEGIES = ["dataparallel", "modelparallel"]
 
 
+GRAVITON_ALLOWED_TARGET_INSTANCE_FAMILY = ["c6g", "t4g", "r6g", "m6g"]
+
+
+GRAVITON_ALLOWED_FRAMEWORKS = set(["tensorflow", "pytorch", "xgboost", "sklearn"])
+
+
 def validate_source_dir(script, directory):
     """Validate that the source directory exists and it contains the user script.
 
@@ -165,12 +171,6 @@ def validate_source_dir(script, directory):
     return True
 
 
-GRAVITON_ALLOWED_TARGET_INSTANCE_FAMILY = ["c6g", "t4g", "r6g", "m6g"]
-
-
-GRAVITON_ALLOWED_FRAMEWORKS = set(["tensorflow", "pytorch"])
-
-
 def validate_source_code_input_against_pipeline_variables(
     entry_point: Optional[Union[str, PipelineVariable]] = None,
     source_dir: Optional[Union[str, PipelineVariable]] = None,