Merge branch 'ChuyangDeng-tf-2-2'

Author: Chuyang Deng

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## v1.59.0 (2020-05-21)
+
+### Features
+
+ * MXNet elastic inference support
+
+### Bug Fixes and Other Changes
+
+ * add Batch Transform data processing options to Airflow config
+ * add v2 warning messages
+ * don't try to use local output path for KMS key in Local Mode
+
+### Documentation Changes
+
+ * add instructions for how to enable 'local code' for Local Mode
+
+## v1.58.4 (2020-05-20)
+
+### Bug Fixes and Other Changes
+
+ * update AutoML default max_candidate value to use the service default
+ * add describe_transform_job in session class
+
+### Documentation Changes
+
+ * clarify support for requirements.txt in Tensorflow docs
+
+### Testing and Release Infrastructure
+
+ * wait for DisassociateTrialComponent to take effect in experiment integ test cleanup
+
+## v1.58.3 (2020-05-19)
+
+### Bug Fixes and Other Changes
+
+ * update DatasetFormat key name for sagemakerCaptureJson
+
+### Documentation Changes
+
+ * update Processing job max_runtime_in_seconds docstring
+
 ## v1.58.2.post0 (2020-05-18)
 
 ### Documentation Changes

README.rst

@@ -205,7 +205,7 @@ By using MXNet SageMaker Estimators, you can train and host MXNet models on Amaz
 
 Supported versions of MXNet: ``0.12.1``, ``1.0.0``, ``1.1.0``, ``1.2.1``, ``1.3.0``, ``1.4.0``, ``1.4.1``, ``1.6.0``.
 
-Supported versions of MXNet for Elastic Inference: ``1.3.0``, ``1.4.0``, ``1.4.1``.
+Supported versions of MXNet for Elastic Inference: ``1.3.0``, ``1.4.0``, ``1.4.1``, ``1.5.1``.
 
 We recommend that you use the latest supported version, because that's where we focus most of our development efforts.
 
@@ -219,7 +219,7 @@ TensorFlow SageMaker Estimators
 
 By using TensorFlow SageMaker Estimators, you can train and host TensorFlow models on Amazon SageMaker.
 
-Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14.0``, ``1.15.0``, ``1.15.2``, ``2.0.0``, ``2.0.1``, ``2.1.0``.
+Supported versions of TensorFlow: ``1.4.1``, ``1.5.0``, ``1.6.0``, ``1.7.0``, ``1.8.0``, ``1.9.0``, ``1.10.0``, ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14.0``, ``1.15.0``, ``1.15.2``, ``2.0.0``, ``2.0.1``, ``2.1.0``, ``2.2.0``.
 
 Supported versions of TensorFlow for Elastic Inference: ``1.11.0``, ``1.12.0``, ``1.13.1``, ``1.14.0``, ``1.15.0``, ``2.0.0``.
 

VERSION

@@ -1 +1 @@
-1.58.3.dev0
+1.59.1.dev0

doc/overview.rst

@@ -732,6 +732,26 @@ You can install all necessary for this feature dependencies using pip:
 
     pip install 'sagemaker[local]' --upgrade
 
+If you want to keep everything local, and not use Amazon S3 either, you can enable "local code" in one of two ways:
+
+- Create a file at ``~/.sagemaker/config.yaml`` that contains:
+
+.. code:: yaml
+
+    local:
+      local_code: true
+
+- Create a ``LocalSession`` and configure it directly:
+
+.. code:: python
+
+    from sagemaker.local import LocalSession
+
+    sagemaker_session = LocalSession()
+    sagemaker_session.config = {'local': {'local_code': True}}
+
+    # pass sagemaker_session to your estimator or model
+
 We can take the example in  `Using Estimators <#using-estimators>`__ , and use either ``local`` or ``local_gpu`` as the instance type.
 
 .. code:: python

doc/using_tf.rst

@@ -20,7 +20,7 @@ For general information about using the SageMaker Python SDK, see :ref:`overview
 
 .. contents::
 
-Supported versions of TensorFlow for Elastic Inference: ``1.11``, ``1.12``, ``1.13``, ``1.14``.
+Supported versions of TensorFlow for Elastic Inference: ``1.11``, ``1.12``, ``1.13``, ``1.14``, ``1.15``, ``2.0``.
 
 
 *****************************
@@ -133,6 +133,34 @@ In your training script the channels will be stored in environment variables ``S
    ``output_path``.
 
 
+Use third-party libraries
+-------------------------
+
+If there are other packages you want to use with your script, you can use a ``requirements.txt`` to install other dependencies at runtime.
+
+For training, support for installing packages using ``requirements.txt`` varies by TensorFlow version as follows:
+
+- For TensorFlow 1.11 or newer using Script Mode without Horovod, TensorFlow 1.15.2 with Python 3.7 or newer, and TensorFlow 2.2 or newer:
+    - Include a ``requirements.txt`` file in the same directory as your training script.
+    - You must specify this directory using the ``source_dir`` argument when creating a TensorFlow estimator.
+- For older versions of TensorFlow using Script Mode with Horovod:
+    - Write a shell script for your entry point that first calls ``pip install -r requirements.txt``, then runs your training script.
+    - For an example of using shell scripts, see `this example notebook <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/tensorflow_script_mode_using_shell_commands/tensorflow_script_mode_using_shell_commands.ipynb>`__.
+- For older versions of TensorFlow using Legacy Mode:
+    - Specify the path to your ``requirements.txt`` file using the ``requirements_file`` argument.
+
+For serving, support for installing packages using ``requirements.txt`` varies by TensorFlow version as follows:
+
+- For TensorFlow 1.11 or newer:
+    - Include a ``requirements.txt`` file in the ``code`` directory.
+- For older versions of TensorFlow:
+    - Specify the path to your ``requirements.txt`` file using the ``SAGEMAKER_REQUIREMENTS`` environment variable.
+
+A ``requirements.txt`` file is a text file that contains a list of items that are installed by using ``pip install``.
+You can also specify the version of an item to install.
+For information about the format of a ``requirements.txt`` file, see `Requirements Files <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`__ in the pip documentation.
+
+
 Create an Estimator
 ===================
 
@@ -215,7 +243,7 @@ Calling ``fit`` starts a SageMaker training job. The training job will execute t
   - starts asynchronous training
 
 If the ``wait=False`` flag is passed to ``fit``, then it returns immediately. The training job continues running
-asynchronously. Later, a Tensorflow estimator can be obtained by attaching to the existing training job.
+asynchronously. Later, a TensorFlow estimator can be obtained by attaching to the existing training job.
 If the training job is not finished, it starts showing the standard output of training and wait until it completes.
 After attaching, the estimator can be deployed as usual.
 
@@ -882,8 +910,7 @@ in the following code:
 You can also bring in external dependencies to help with your data
 processing. There are 2 ways to do this:
 
-1. If you included ``requirements.txt`` in your ``source_dir`` or in
-   your dependencies, the container installs the Python dependencies at runtime using ``pip install -r``:
+1. If your model archive contains ``code/requirements.txt``, the container will install the Python dependencies at runtime using ``pip install -r``.
 
 .. code::
 

src/sagemaker/__init__.py

@@ -13,6 +13,8 @@
 """Placeholder docstring"""
 from __future__ import absolute_import
 
+import logging
+import sys
 import importlib_metadata
 
 from sagemaker import estimator, parameter, tuner  # noqa: F401
@@ -61,3 +63,10 @@
 from sagemaker.automl.candidate_estimator import CandidateEstimator, CandidateStep  # noqa: F401
 
 __version__ = importlib_metadata.version("sagemaker")
+
+if sys.version[0] == "2":
+    logging.getLogger("sagemaker").warning(
+        "SageMaker Python SDK v2 will no longer support Python 2. "
+        "Please see https://github.com/aws/sagemaker-python-sdk/issues/1459 "
+        "for more information"
+    )

src/sagemaker/amazon/amazon_estimator.py

@@ -616,6 +616,11 @@ def get_image_uri(region_name, repo_name, repo_version=1):
         repo_name:
         repo_version:
     """
+    logger.warning(
+        "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class "
+        "in SageMaker Python SDK v2."
+    )
+
     if repo_name == "xgboost":
         if not _is_latest_xgboost_version(repo_version):
             logging.warning(

src/sagemaker/automl/automl.py

@@ -39,7 +39,7 @@ def __init__(
         encrypt_inter_container_traffic=False,
         vpc_config=None,
         problem_type=None,
-        max_candidates=500,
+        max_candidates=None,
         max_runtime_per_training_job_in_seconds=None,
         total_job_runtime_in_seconds=None,
         job_objective=None,

src/sagemaker/estimator.py

@@ -39,6 +39,7 @@
     UploadedCode,
     validate_source_dir,
     _region_supports_debugger,
+    parameter_v2_rename_warning,
 )
 from sagemaker.job import _Job
 from sagemaker.local import LocalSession
@@ -1273,6 +1274,7 @@ def __init__(
                 https://docs.aws.amazon.com/sagemaker/latest/dg/API_AlgorithmSpecification.html#SageMaker-Type-AlgorithmSpecification-EnableSageMakerMetricsTimeSeries
                 (default: ``None``).
         """
+        logging.warning(parameter_v2_rename_warning("image_name", "image_uri"))
         self.image_name = image_name
         self.hyperparam_dict = hyperparameters.copy() if hyperparameters else {}
         super(Estimator, self).__init__(
@@ -1637,6 +1639,8 @@ def __init__(
         self.container_log_level = container_log_level
         self.code_location = code_location
         self.image_name = image_name
+        if image_name is not None:
+            logging.warning(parameter_v2_rename_warning("image_name", "image_uri"))
 
         self.uploaded_code = None
 
@@ -1722,11 +1726,14 @@ def _stage_user_code_in_s3(self):
             code_bucket = self.sagemaker_session.default_bucket()
             code_s3_prefix = "{}/{}".format(self._current_job_name, "source")
             kms_key = None
-
         elif self.code_location is None:
             code_bucket, _ = parse_s3_url(self.output_path)
             code_s3_prefix = "{}/{}".format(self._current_job_name, "source")
             kms_key = self.output_kms_key
+        elif local_mode:
+            code_bucket, key_prefix = parse_s3_url(self.code_location)
+            code_s3_prefix = "/".join(filter(None, [key_prefix, self._current_job_name, "source"]))
+            kms_key = None
         else:
             code_bucket, key_prefix = parse_s3_url(self.code_location)
             code_s3_prefix = "/".join(filter(None, [key_prefix, self._current_job_name, "source"]))

src/sagemaker/fw_utils.py

@@ -34,7 +34,10 @@
 instantiated with positional or keyword arguments.
 """
 
-EMPTY_FRAMEWORK_VERSION_WARNING = "No framework_version specified, defaulting to version {}."
+EMPTY_FRAMEWORK_VERSION_WARNING = (
+    "No framework_version specified, defaulting to version {}. "
+    "framework_version will be required in SageMaker Python SDK v2."
+)
 LATER_FRAMEWORK_VERSION_WARNING = (
     "This is not the latest supported version. "
     "If you would like to use version {latest}, "
@@ -52,6 +55,10 @@
     "fully leverage all GPU cores; the parameter server will be configured to run "
     "only one worker per host regardless of the number of GPUs."
 )
+PARAMETER_V2_RENAME_WARNING = (
+    "Parameter {v1_parameter_name} will be renamed to {v2_parameter_name} "
+    "in SageMaker Python SDK v2."
+)
 
 
 EMPTY_FRAMEWORK_VERSION_ERROR = (
@@ -253,6 +260,11 @@ def create_image_uri(
     Returns:
         str: The appropriate image URI based on the given parameters.
     """
+    logger.warning(
+        "'create_image_uri' will be deprecated in favor of 'ImageURIProvider' class "
+        "in SageMaker Python SDK v2."
+    )
+
     optimized_families = optimized_families or []
 
     if py_version and py_version not in VALID_PY_VERSIONS:
@@ -647,6 +659,17 @@ def python_deprecation_warning(framework, latest_supported_version):
     )
 
 
+def parameter_v2_rename_warning(v1_parameter_name, v2_parameter_name):
+    """
+    Args:
+        v1_parameter_name: parameter name used in SageMaker Python SDK v1
+        v2_parameter_name: parameter name used in SageMaker Python SDK v2
+    """
+    return PARAMETER_V2_RENAME_WARNING.format(
+        v1_parameter_name=v1_parameter_name, v2_parameter_name=v2_parameter_name
+    )
+
+
 def _region_supports_debugger(region_name):
     """Returns boolean indicating whether the region supports Amazon SageMaker Debugger.
 

src/sagemaker/inputs.py

@@ -13,9 +13,13 @@
 """Amazon SageMaker channel configurations for S3 data sources and file system data sources"""
 from __future__ import absolute_import, print_function
 
+import logging
+
 FILE_SYSTEM_TYPES = ["FSxLustre", "EFS"]
 FILE_SYSTEM_ACCESS_MODES = ["ro", "rw"]
 
+logger = logging.getLogger("sagemaker")
+
 
 class s3_input(object):
     """Amazon SageMaker channel configurations for S3 data sources.
@@ -76,6 +80,9 @@ def __init__(
                 this channel. See the SageMaker API documentation for more info:
                 https://docs.aws.amazon.com/sagemaker/latest/dg/API_ShuffleConfig.html
         """
+        logger.warning(
+            "'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2."
+        )
 
         self.config = {
             "DataSource": {"S3DataSource": {"S3DataType": s3_data_type, "S3Uri": s3_data}}

src/sagemaker/model.py

@@ -108,6 +108,8 @@ def __init__(
             model_kms_key (str): KMS key ARN used to encrypt the repacked
                 model archive file if the model is repacked
         """
+        LOGGER.warning(fw_utils.parameter_v2_rename_warning("image", "image_uri"))
+
         self.model_data = model_data
         self.image = image
         self.role = role

src/sagemaker/mxnet/README.rst

@@ -6,7 +6,7 @@ With the SageMaker Python SDK, you can train and host MXNet models on Amazon Sag
 
 Supported versions of MXNet: ``0.12.1``, ``1.0.0``, ``1.1.0``, ``1.2.1``, ``1.3.0``, ``1.4.0``, ``1.4.1``, ``1.6.0``.
 
-Supported versions of MXNet for Elastic Inference: ``1.3.0``, ``1.4.0``, ``1.4.1``.
+Supported versions of MXNet for Elastic Inference: ``1.3.0``, ``1.4.0``, ``1.4.1``, ``1.5.1``.
 
 Supported versions of MXNet for Inferentia: ``1.5.1``.
 

src/sagemaker/mxnet/estimator.py

@@ -21,6 +21,7 @@
     framework_version_from_tag,
     empty_framework_version_warning,
     python_deprecation_warning,
+    parameter_v2_rename_warning,
     is_version_equal_or_higher,
     warn_if_parameter_server_with_multi_gpu,
 )
@@ -129,6 +130,7 @@ def __init__(
             )
 
         if distributions is not None:
+            logger.warning(parameter_v2_rename_warning("distributions", "distribution"))
             train_instance_type = kwargs.get("train_instance_type")
             warn_if_parameter_server_with_multi_gpu(
                 training_instance_type=train_instance_type, distributions=distributions