Merge branch 'zwei' into add-csv-deserializer

Author: Balaji Veeramani

CHANGELOG.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## v2.0.0.rc1 (2020-07-08)
+
+### Breaking Changes
+
+ * Move StreamDeserializer to sagemaker.deserializers
+ * Move StringDeserializer to sagemaker.deserializers
+ * rename record_deserializer to RecordDeserializer
+ * remove "train_" where redundant in parameter/variable names
+ * Add BytesDeserializer
+ * rename image to image_uri
+ * rename image_name to image_uri
+ * create new inference resources during model.deploy() and model.transformer()
+ * rename session parameter to sagemaker_session in S3 utility classes
+ * rename distributions to distribution in TF/MXNet estimators
+ * deprecate update_endpoint arg in deploy()
+ * create new inference resources during estimator.deploy() or estimator.transformer()
+ * deprecate delete_endpoint() for estimators and HyperparameterTuner
+ * refactor Predictor attribute endpoint to endpoint_name
+ * make instance_type optional for Airflow model configs
+ * refactor name of RealTimePredictor to Predictor
+ * remove check for Python 2 string in sagemaker.predictor._is_sequence_like()
+ * deprecate sagemaker.utils.to_str()
+ * drop Python 2 support
+
+### Features
+
+ * add BaseSerializer and BaseDeserializer
+ * add Predictor.update_endpoint()
+
+### Bug Fixes and Other Changes
+
+ * handle "train_*" renames in v2 migration tool
+ * handle image_uri rename for Session methods in v2 migration tool
+ * Update BytesDeserializer accept header
+ * handle image_uri rename for estimators and models in v2 migration tool
+ * handle image_uri rename in Airflow model config functions in v2 migration tool
+ * update migration tool for S3 utility functions
+ * set _current_job_name and base_tuning_job_name in HyperparameterTuner.attach()
+ * infer base name from job name in estimator.attach()
+ * ensure generated names are < 63 characters when deploying compiled models
+ * add TF migration documentation to error message
+
+### Documentation Changes
+
+ * update documentation with v2.0.0.rc1 changes
+ * remove 'train_*' prefix from estimator parameters
+ * update documentation for image_name/image --> image_uri
+
+### Testing and Release Infrastructure
+
+ * refactor matching logic in v2 migration tool
+ * add cli modifier for RealTimePredictor and derived classes
+ * change coverage settings to reduce intermittent errors
+ * clean up pickle.load logic in integ tests
+ * use fixture for Python version in framework integ tests
+ * remove assumption of Python 2 unit test runs
+
 ## v1.68.0 (2020-07-07)
 
 ### Features

VERSION

@@ -1 +1 @@
-2.0.0.rc0
+2.0.0.rc1

bin/sagemaker-submit

@@ -56,4 +56,4 @@ if __name__ == '__main__':
                                            hyperparameters=hyperparameters,
                                            instance_count=args.instance_count,
                                            instance_type=args.instance_type)
-    estimator.fit(sagemaker.s3_input(args.data))
+    estimator.fit(sagemaker.TrainingInput(args.data))

doc/algorithms/linear_learner.rst

@@ -8,7 +8,7 @@ The Amazon SageMaker LinearLearner algorithm.
     :undoc-members:
     :show-inheritance:
     :inherited-members:
-    :exclude-members: image_uri, train_instance_count, train_instance_type, predictor_type, binary_classifier_model_selection_criteria, target_recall, target_precision, positive_example_weight_mult, epochs, use_bias, num_models, parameter, num_calibration_samples, calibration, init_method, init_scale, init_sigma, init_bias, optimizer, loss, wd, l1, momentum, learning_rate, beta_1, beta_2, bias_lr_mult, use_lr_scheduler, lr_scheduler_step, lr_scheduler_factor, lr_scheduler_minimum_lr, lr_scheduler_minimum_lr, mini_batch_size, feature_dim, bias_wd_mult, MAX_DEFAULT_BATCH_SIZE
+    :exclude-members: image_uri, instance_count, instance_type, predictor_type, binary_classifier_model_selection_criteria, target_recall, target_precision, positive_example_weight_mult, epochs, use_bias, num_models, parameter, num_calibration_samples, calibration, init_method, init_scale, init_sigma, init_bias, optimizer, loss, wd, l1, momentum, learning_rate, beta_1, beta_2, bias_lr_mult, use_lr_scheduler, lr_scheduler_step, lr_scheduler_factor, lr_scheduler_minimum_lr, lr_scheduler_minimum_lr, mini_batch_size, feature_dim, bias_wd_mult, MAX_DEFAULT_BATCH_SIZE
 
 .. autoclass:: sagemaker.LinearLearnerModel
     :members:

doc/amazon_sagemaker_debugger.rst

@@ -54,8 +54,8 @@ The ``DebuggerHookConfig`` accepts one or more objects of type ``CollectionConfi
 
     estimator = TensorFlow(
         role=role,
-        train_instance_count=1,
-        train_instance_type=train_instance_type,
+        instance_count=1,
+        instance_type=instance_type,
         debugger_hook_config=debugger_hook_config
     )
 
@@ -215,8 +215,8 @@ Sample Usages
 
     estimator = TensorFlow(
             role=role,
-            train_instance_count=1,
-            train_instance_type=train_instance_type,
+            instance_count=1,
+            instance_type=instance_type,
             rules=[Rule.sagemaker(vanishing_gradient())]
     )
 
@@ -232,8 +232,8 @@ In the example above, Amazon SageMaker pulls the collection configuration best s
 
     estimator = TensorFlow(
             role=role,
-            train_instance_count=1,
-            train_instance_type=train_instance_type,
+            instance_count=1,
+            instance_type=instance_type,
             rules=[Rule.sagemaker(vanishing_gradient()), Rule.sagemaker(weight_update_ratio())]
     )
 
@@ -269,8 +269,8 @@ Here we modify the ``weight_update_ratio`` rule to store a custom collection rat
 
     estimator = TensorFlow(
             role=role,
-            train_instance_count=1,
-            train_instance_type=train_instance_type,
+            instance_count=1,
+            instance_type=instance_type,
             rules=[
                 Rule.sagemaker(vanishing_gradient()),
                 wur_with_customization
@@ -317,8 +317,8 @@ To evaluate the custom rule against the training:
 
     estimator = TensorFlow(
         role=role,
-        train_instance_count=1,
-        train_instance_type=train_instance_type,
+        instance_count=1,
+        instance_type=instance_type,
         rules=[
             custom_gradient_rule
         ]
@@ -344,8 +344,8 @@ To enable the debugging hook to emit TensorBoard data, you need to specify the n
 
     estimator = TensorFlow(
         role=role,
-        train_instance_count=1,
-        train_instance_type=train_instance_type,
+        instance_count=1,
+        instance_type=instance_type,
         tensorboard_output_config=tensorboard_output_config
     )
 
@@ -392,8 +392,8 @@ To disable the hook initialization, you can do so by specifying ``False`` for va
 
     estimator = TensorFlow(
         role=role,
-        train_instance_count=1,
-        train_instance_type=train_instance_type,
+        instance_count=1,
+        instance_type=instance_type,
         debugger_hook_config=False
     )
 

doc/frameworks/chainer/using_chainer.rst

@@ -138,8 +138,8 @@ directories ('train' and 'test').
 .. code:: python
 
     chainer_estimator = Chainer('chainer-train.py',
-                                train_instance_type='ml.p3.2xlarge',
-                                train_instance_count=1,
+                                instance_type='ml.p3.2xlarge',
+                                instance_count=1,
                                 framework_version='5.0.0',
                                 py_version='py3',
                                 hyperparameters = {'epochs': 20, 'batch-size': 64, 'learning-rate': 0.1})
@@ -191,7 +191,7 @@ Chainer allows you to train a model on multiple nodes using ChainerMN_, which di
 In order to run distributed Chainer training on SageMaker, your training script should use a ``chainermn`` Communicator
 object to coordinate training between multiple hosts.
 
-SageMaker runs your script with ``mpirun`` if ``train_instance_count`` is greater than two.
+SageMaker runs your script with ``mpirun`` if ``instance_count`` is greater than two.
 The following are optional arguments modify how MPI runs your distributed training script.
 
 -  ``use_mpi`` Boolean that overrides whether to run your training script with MPI.
@@ -221,8 +221,8 @@ operation.
 
     # Train my estimator
     chainer_estimator = Chainer(entry_point='train_and_deploy.py',
-                                train_instance_type='ml.p3.2xlarge',
-                                train_instance_count=1,
+                                instance_type='ml.p3.2xlarge',
+                                instance_count=1,
                                 framework_version='5.0.0',
                                 py_version='py3')
     chainer_estimator.fit('s3://my_bucket/my_training_data/')

doc/frameworks/mxnet/using_mxnet.rst

@@ -181,8 +181,8 @@ The following code sample shows how you train a custom MXNet script "train.py".
 .. code:: python
 
     mxnet_estimator = MXNet('train.py',
-                            train_instance_type='ml.p2.xlarge',
-                            train_instance_count=1,
+                            instance_type='ml.p2.xlarge',
+                            instance_count=1,
                             framework_version='1.6.0',
                             py_version='py3',
                             hyperparameters={'batch-size': 100,
@@ -233,8 +233,8 @@ If you use the ``MXNet`` estimator to train the model, you can call ``deploy`` t
     mxnet_estimator = MXNet('train.py',
                             framework_version='1.6.0',
                             py_version='py3',
-                            train_instance_type='ml.p2.xlarge',
-                            train_instance_count=1)
+                            instance_type='ml.p2.xlarge',
+                            instance_count=1)
     mxnet_estimator.fit('s3://my_bucket/my_training_data/')
 
     # Deploy my estimator to an Amazon SageMaker Endpoint and get a Predictor

doc/frameworks/pytorch/using_pytorch.rst

@@ -152,8 +152,8 @@ directories ('train' and 'test').
 .. code:: python
 
     pytorch_estimator = PyTorch('pytorch-train.py',
-                                train_instance_type='ml.p3.2xlarge',
-                                train_instance_count=1,
+                                instance_type='ml.p3.2xlarge',
+                                instance_count=1,
                                 framework_version='1.5.0',
                                 py_version='py3',
                                 hyperparameters = {'epochs': 20, 'batch-size': 64, 'learning-rate': 0.1})
@@ -201,7 +201,7 @@ Distributed PyTorch Training
 ============================
 
 You can run a multi-machine, distributed PyTorch training using the PyTorch Estimator. By default, PyTorch objects will
-submit single-machine training jobs to SageMaker. If you set ``train_instance_count`` to be greater than one, multi-machine
+submit single-machine training jobs to SageMaker. If you set ``instance_count`` to be greater than one, multi-machine
 training jobs will be launched when ``fit`` is called. When you run multi-machine training, SageMaker will import your
 training script and run it on each host in the cluster.
 
@@ -246,8 +246,8 @@ operation.
 
     # Train my estimator
     pytorch_estimator = PyTorch(entry_point='train_and_deploy.py',
-                                train_instance_type='ml.p3.2xlarge',
-                                train_instance_count=1,
+                                instance_type='ml.p3.2xlarge',
+                                instance_count=1,
                                 framework_version='1.5.0',
                                 py_version='py3')
     pytorch_estimator.fit('s3://my_bucket/my_training_data/')

doc/frameworks/rl/using_rl.rst

@@ -31,8 +31,8 @@ You can then create an ``RLEstimator`` with keyword arguments to point to this s
                                toolkit_version='0.11.1',
                                framework=RLFramework.TENSORFLOW,
                                role='SageMakerRole',
-                               train_instance_type='ml.p3.2xlarge',
-                               train_instance_count=1)
+                               instance_type='ml.p3.2xlarge',
+                               instance_count=1)
 
 After that, you simply tell the estimator to start a training job:
 
@@ -81,9 +81,9 @@ these in the constructor, either positionally or as keyword arguments.
    endpoints use this role to access training data and model artifacts.
    After the endpoint is created, the inference code might use the IAM
    role, if accessing AWS resource.
--  ``train_instance_count`` Number of Amazon EC2 instances to use for
+-  ``instance_count`` Number of Amazon EC2 instances to use for
    training.
--  ``train_instance_type`` Type of EC2 instance to use for training, for
+-  ``instance_type`` Type of EC2 instance to use for training, for
    example, 'ml.m4.xlarge'.
 
 You must as well include either:
@@ -158,8 +158,8 @@ In case if ``image_uri`` was specified it would use provided image for the deplo
                                toolkit_version='0.11.0',
                                framework=RLFramework.MXNET,
                                role='SageMakerRole',
-                               train_instance_type='ml.c4.2xlarge',
-                               train_instance_count=1)
+                               instance_type='ml.c4.2xlarge',
+                               instance_count=1)
 
     rl_estimator.fit()
 

doc/frameworks/sklearn/using_sklearn.rst

@@ -134,7 +134,7 @@ directories ('train' and 'test').
 .. code:: python
 
     sklearn_estimator = SKLearn('sklearn-train.py',
-                                train_instance_type='ml.m4.xlarge',
+                                instance_type='ml.m4.xlarge',
                                 framework_version='0.20.0',
                                 hyperparameters = {'epochs': 20, 'batch-size': 64, 'learning-rate': 0.1})
     sklearn_estimator.fit({'train': 's3://my-data-bucket/path/to/my/training/data',
@@ -198,7 +198,7 @@ operation.
 
     # Train my estimator
     sklearn_estimator = SKLearn(entry_point='train_and_deploy.py',
-                                train_instance_type='ml.m4.xlarge',
+                                instance_type='ml.m4.xlarge',
                                 framework_version='0.20.0')
     sklearn_estimator.fit('s3://my_bucket/my_training_data/')
 

doc/frameworks/tensorflow/deploying_tensorflow_serving.rst

@@ -22,17 +22,21 @@ estimator object to create a SageMaker Endpoint:
 
   from sagemaker.tensorflow import TensorFlow
 
-  estimator = TensorFlow(entry_point='tf-train.py', ..., train_instance_count=1,
-                         train_instance_type='ml.c4.xlarge', framework_version='1.11')
+  estimator = TensorFlow(
+      entry_point="tf-train.py",
+      ...,
+      instance_count=1,
+      instance_type="ml.c4.xlarge",
+      framework_version="2.2",
+      py_version="py37",
+  )
 
   estimator.fit(inputs)
 
-  predictor = estimator.deploy(initial_instance_count=1,
-                               instance_type='ml.c5.xlarge',
-                               endpoint_type='tensorflow-serving')
+  predictor = estimator.deploy(initial_instance_count=1, instance_type="ml.c5.xlarge")
 
 
-The code block above deploys a SageMaker Endpoint with one instance of the type 'ml.c5.xlarge'.
+The code block above deploys a SageMaker Endpoint with one instance of the type "ml.c5.xlarge".
 
 What happens when deploy is called
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/frameworks/tensorflow/upgrade_from_legacy.rst

@@ -105,7 +105,7 @@ the difference in code would be as follows:
         source_dir="code",
         framework_version="1.10.0",
         py_version="py2",
-        train_instance_type="ml.m4.xlarge",
+        instance_type="ml.m4.xlarge",
         image_uri="520713654638.dkr.ecr.us-west-2.amazonaws.com/sagemaker-tensorflow:1.10.0-cpu-py2",
         hyperparameters={
             "training_steps": 100,