Merge remote-tracking branch 'public/master' into pytorch-release

Author: nadiaya

CHANGELOG.rst

@@ -2,6 +2,12 @@
 CHANGELOG
 =========
 
+1.4.2dev
+========
+
+* bug-fix: Unit Tests: Improve unit test runtime
+* bug-fix: Estimators: Fix attach for LDA
+
 1.4.1
 =====
 
@@ -18,6 +24,7 @@ CHANGELOG
 * feature: Analytics: Add functions for metrics in Training and Hyperparameter Tuning jobs
 * feature: Estimators: add support for tagging training jobs
 
+
 1.3.0
 =====
 

README.rst

@@ -30,7 +30,8 @@ Table of Contents
 5. `Chainer SageMaker Estimators <#chainer-sagemaker-estimators>`__
 6. `AWS SageMaker Estimators <#aws-sagemaker-estimators>`__
 7. `BYO Docker Containers with SageMaker Estimators <#byo-docker-containers-with-sagemaker-estimators>`__
-8. `BYO Model <#byo-model>`__
+8. `SageMaker Automatic Model Tuning <#sagemaker-automatic-model-tuning>`__
+9. `BYO Model <#byo-model>`__
 
 
 Getting SageMaker Python SDK
@@ -263,6 +264,86 @@ Please refer to the full example in the examples repo:
 The example notebook is is located here:
 ``advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb``
 
+
+SageMaker Automatic Model Tuning
+--------------------------------
+
+All of the estimators can be used with SageMaker Automatic Model Tuning, which performs hyperparameter tuning jobs.
+A hyperparameter tuning job runs multiple training jobs that differ by the values of their hyperparameters to find the best training job.
+It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose.
+If you're not using an Amazon ML algorithm, then the metric is defined by a regular expression (regex) you provide for going through the training job's logs.
+You can read more about SageMaker Automatic Model Tuning in the `AWS documentation <https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html>`__.
+
+The SageMaker Python SDK contains a ``HyperparameterTuner`` class for creating and interacting with hyperparameter training jobs.
+Here is a basic example of how to use it:
+
+.. code:: python
+
+    from sagemaker.tuner import HyperparameterTuner, ContinuousParameter
+
+    # Configure HyperparameterTuner
+    my_tuner = HyperparameterTuner(estimator=my_estimator,  # previously-configured Estimator object
+                                   objective_metric_name='validation-accuracy',
+                                   hyperparameter_ranges={'learning-rate': ContinuousParameter(0.05, 0.06)},
+                                   metric_definitions=[{'Name': 'validation-accuracy', 'Regex': 'validation-accuracy=(\d\.\d+)'}],
+                                   max_jobs=100,
+                                   max_parallel_jobs=10)
+
+    # Start hyperparameter tuning job
+    my_tuner.fit({'train': 's3://my_bucket/my_training_data', 'test': 's3://my_bucket_my_testing_data'})
+
+    # Deploy best model
+    my_predictor = my_tuner.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')
+
+    # Make a prediction against the SageMaker endpoint
+    response = my_predictor.predict(my_prediction_data)
+
+    # Tear down the SageMaker endpoint
+    my_tuner.delete_endpoint()
+
+This example shows a hyperparameter tuning job that creates up to 100 training jobs, running up to 10 at a time.
+Each training job's learning rate will be a value between 0.05 and 0.06, but this value will differ between training jobs.
+You can read more about how these values are chosen in the `AWS documentation <https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-how-it-works.html>`__.
+
+A hyperparameter range can be one of three types: continuous, integer, or categorical.
+The SageMaker Python SDK provides corresponding classes for defining these different types.
+You can define up to 20 hyperparameters to search over, but each value of a categorical hyperparameter range counts against that limit.
+
+If you are using an Amazon ML algorithm, you don't need to pass in anything for ``metric_definitions``.
+In addition, the ``fit()`` call uses a list of ``RecordSet`` objects instead of a dictionary:
+
+.. code:: python
+
+    # Create RecordSet object for each data channel
+    train_records = RecordSet(...)
+    test_records = RecordSet(...)
+
+    # Start hyperparameter tuning job
+    my_tuner.fit([train_records, test_records])
+
+There is also an analytics object associated with each ``HyperparameterTuner`` instance that presents useful information about the hyperparameter tuning job.
+For example, the ``dataframe`` method gets a pandas dataframe summarizing the associated training jobs:
+
+.. code:: python
+
+    # Retrieve analytics object
+    my_tuner_analytics = my_tuner.analytics()
+
+    # Look at summary of associated training jobs
+    my_dataframe = my_tuner_analytics.dataframe()
+
+For more detailed examples of running hyperparameter tuning jobs, see:
+
+- `Using the TensorFlow estimator with hyperparameter tuning <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/tensorflow_mnist/hpo_tensorflow_mnist.ipynb>`__
+- `Bringing your own estimator for hyperparameter tuning <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/r_bring_your_own/hpo_r_bring_your_own.ipynb>`__
+- `Analyzing results <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/hyperparameter_tuning/analyze_results/HPO_Analyze_TuningJob_Results.ipynb>`__
+
+For more detailed explanations of the classes that this library provides for automatic model tuning, see:
+
+- `API docs for HyperparameterTuner and parameter range classes <https://sagemaker.readthedocs.io/en/latest/tuner.html>`__
+- `API docs for analytics classes <https://sagemaker.readthedocs.io/en/latest/analytics.html>`__
+
+
 FAQ
 ---
 

doc/analytics.rst

@@ -0,0 +1,17 @@
+Analytics
+---------
+
+.. autoclass:: sagemaker.analytics.AnalyticsMetricsBase
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.analytics.HyperparameterTuningJobAnalytics
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.analytics.TrainingJobAnalytics
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/index.rst

@@ -4,7 +4,7 @@ Amazon SageMaker Python SDK is an open source library for training and deploying
 
 With the SDK, you can train and deploy models using popular deep learning frameworks: **Apache MXNet** and **TensorFlow**. You can also train and deploy models with **algorithms provided by Amazon**, these are scalable implementations of core machine learning algorithms that are optimized for SageMaker and GPU training. If you have **your own algorithms** built into SageMaker-compatible Docker containers, you can train and host models using these as well.
 
-Here you'll find API docs for SageMaker Python SDK. The project home-page is in Github: https://github.com/aws/sagemaker-python-sdk, there you can find the SDK source, installation instructions and a general overview of the library there. 
+Here you'll find API docs for SageMaker Python SDK. The project home-page is in Github: https://github.com/aws/sagemaker-python-sdk, there you can find the SDK source, installation instructions and a general overview of the library there.
 
 Overview
 ----------
@@ -14,9 +14,11 @@ The SageMaker Python SDK consists of a few primary interfaces:
     :maxdepth: 2
 
     estimators
+    tuner
     predictors
     session
     model
+    analytics
 
 MXNet
 ----------

doc/tuner.rst

@@ -0,0 +1,22 @@
+HyperparameterTuner
+-------------------
+
+.. autoclass:: sagemaker.tuner.HyperparameterTuner
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.tuner.ContinuousParameter
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.tuner.IntegerParameter
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. autoclass:: sagemaker.tuner.CategoricalParameter
+    :members:
+    :undoc-members:
+    :show-inheritance:

src/sagemaker/amazon/lda.py

@@ -78,8 +78,10 @@ def __init__(self, role, train_instance_type, num_topics,
             tol (float): Optional. Target error tolerance for the ALS phase of the algorithm.
             **kwargs: base class keyword argument values.
         """
-
         # this algorithm only supports single instance training
+        if kwargs.pop('train_instance_count', 1) != 1:
+            print('LDA only supports single instance training. Defaulting to 1 {}.'.format(train_instance_type))
+
         super(LDA, self).__init__(role, 1, train_instance_type, **kwargs)
         self.num_topics = num_topics
         self.alpha0 = alpha0

src/sagemaker/analytics.py

@@ -64,25 +64,24 @@ def _fetch_dataframe(self):
         pass
 
     def clear_cache(self):
-        """Clears the object of all local caches of API methods, so
+        """Clear the object of all local caches of API methods, so
         that the next time any properties are accessed they will be refreshed from
         the service.
         """
         self._dataframe = None
 
 
 class HyperparameterTuningJobAnalytics(AnalyticsMetricsBase):
-    """Fetches results about this tuning job and makes them accessible for analytics.
+    """Fetch results about a hyperparameter tuning job and make them accessible for analytics.
     """
 
     def __init__(self, hyperparameter_tuning_job_name, sagemaker_session=None):
-        """Initialize an ``HyperparameterTuningJobAnalytics`` instance.
+        """Initialize a ``HyperparameterTuningJobAnalytics`` instance.
 
         Args:
-            hyperparameter_tuning_job_name (str): name of the HyperparameterTuningJob to
-                analyze.
+            hyperparameter_tuning_job_name (str): name of the HyperparameterTuningJob to analyze.
             sagemaker_session (sagemaker.session.Session): Session object which manages interactions with
-                Amazon SageMaker APIs and any other AWS services needed. If not specified, the estimator creates one
+                Amazon SageMaker APIs and any other AWS services needed. If not specified, one is created
                 using the default AWS configuration chain.
         """
         sagemaker_session = sagemaker_session or Session()
@@ -100,16 +99,16 @@ def __repr__(self):
         return "<sagemaker.HyperparameterTuningJobAnalytics for %s>" % self.name
 
     def clear_cache(self):
-        """Clears the object of all local caches of API methods.
+        """Clear the object of all local caches of API methods.
         """
         super(HyperparameterTuningJobAnalytics, self).clear_cache()
         self._tuning_job_describe_result = None
         self._training_job_summaries = None
 
     def _fetch_dataframe(self):
-        """Returns a pandas dataframe with all the training jobs, their
-        hyperparameters, results, and metadata about the training jobs.
-        Includes a column to indicate that any job was the best seen so far.
+        """Return a pandas dataframe with all the training jobs, along with their
+        hyperparameters, results, and metadata. This also includes a column to indicate
+        if a training job was the best seen so far.
         """
         def reshape(training_summary):
             # Helper method to reshape a single training job summary into a dataframe record
@@ -139,8 +138,8 @@ def reshape(training_summary):
 
     @property
     def tuning_ranges(self):
-        """A dict describing the ranges of all tuned hyperparameters.
-        Dict's key is the name of the hyper param.  Dict's value is the range.
+        """A dictionary describing the ranges of all tuned hyperparameters.
+        The keys are the names of the hyperparameter, and the values are the ranges.
         """
         out = {}
         for _, ranges in self.description()['HyperParameterTuningJobConfig']['ParameterRanges'].items():
@@ -149,10 +148,13 @@ def tuning_ranges(self):
         return out
 
     def description(self, force_refresh=False):
-        """Response to DescribeHyperParameterTuningJob
+        """Call ``DescribeHyperParameterTuningJob`` for the hyperparameter tuning job.
 
         Args:
             force_refresh (bool): Set to True to fetch the latest data from SageMaker API.
+
+        Returns:
+            dict: The Amazon SageMaker response for ``DescribeHyperParameterTuningJob``.
         """
         if force_refresh:
             self.clear_cache()
@@ -163,10 +165,13 @@ def description(self, force_refresh=False):
         return self._tuning_job_describe_result
 
     def training_job_summaries(self, force_refresh=False):
-        """A list of everything (paginated) from ListTrainingJobsForTuningJob
+        """A (paginated) list of everything from ``ListTrainingJobsForTuningJob``.
 
         Args:
             force_refresh (bool): Set to True to fetch the latest data from SageMaker API.
+
+        Returns:
+            dict: The Amazon SageMaker response for ``ListTrainingJobsForTuningJob``.
         """
         if force_refresh:
             self.clear_cache()
@@ -191,19 +196,19 @@ def training_job_summaries(self, force_refresh=False):
 
 
 class TrainingJobAnalytics(AnalyticsMetricsBase):
-    """Fetches training curve data from CloudWatch Metrics for a specific training job.
+    """Fetch training curve data from CloudWatch Metrics for a specific training job.
     """
 
     CLOUDWATCH_NAMESPACE = '/aws/sagemaker/HyperParameterTuningJobs'
 
     def __init__(self, training_job_name, metric_names, sagemaker_session=None):
-        """Initialize an ``TrainingJobAnalytics`` instance.
+        """Initialize a ``TrainingJobAnalytics`` instance.
 
         Args:
             training_job_name (str): name of the TrainingJob to analyze.
             metric_names (list): string names of all the metrics to collect for this training job
             sagemaker_session (sagemaker.session.Session): Session object which manages interactions with
-                Amazon SageMaker APIs and any other AWS services needed. If not specified, the estimator creates one
+                Amazon SageMaker APIs and any other AWS services needed. If not specified, one is specified
                 using the default AWS configuration chain.
         """
         sagemaker_session = sagemaker_session or Session()
@@ -223,7 +228,7 @@ def __repr__(self):
         return "<sagemaker.TrainingJobAnalytics for %s>" % self.name
 
     def clear_cache(self):
-        """Clears the object of all local caches of API methods, so
+        """Clear the object of all local caches of API methods, so
         that the next time any properties are accessed they will be refreshed from
         the service.
         """
@@ -232,7 +237,7 @@ def clear_cache(self):
         self._time_interval = self._determine_timeinterval()
 
     def _determine_timeinterval(self):
-        """Returns a dict with two datetime objects, start_time and end_time
+        """Return a dictionary with two datetime objects, start_time and end_time,
         covering the interval of the training job
         """
         description = self._sage_client.describe_training_job(TrainingJobName=self.name)
@@ -249,7 +254,7 @@ def _fetch_dataframe(self):
         return pd.DataFrame(self._data)
 
     def _fetch_metric(self, metric_name):
-        """Fetches all the values of a named metric, and adds them to _data
+        """Fetch all the values of a named metric, and add them to _data
         """
         request = {
             'Namespace': self.CLOUDWATCH_NAMESPACE,
@@ -284,7 +289,7 @@ def _fetch_metric(self, metric_name):
             self._add_single_metric(elapsed_seconds, metric_name, value)
 
     def _add_single_metric(self, timestamp, metric_name, value):
-        """Stores a single metric in the _data dict which can be
+        """Store a single metric in the _data dict which can be
         converted to a dataframe.
         """
         # note that this method is built this way to make it possible to

src/sagemaker/estimator.py

@@ -319,7 +319,7 @@ def delete_endpoint(self):
 
     @property
     def training_job_analytics(self):
-        """Returns a TrainingJobAnalytics object for the current training job.
+        """Return a ``TrainingJobAnalytics`` object for the current training job.
         """
         if self._current_job_name is None:
             raise ValueError('Estimator is not associated with a TrainingJob')