Merge branch 'master' into master

Author: djarpin

README.md

@@ -13,6 +13,7 @@ These examples provide a gentle introduction to machine learning concepts as the
 - [Time-series Forecasting](introduction_to_applying_machine_learning/linear_time_series_forecast) generates a forecast for topline product demand using Amazon SageMaker's Linear Learner algorithm.
 - [Cancer Prediction](introduction_to_applying_machine_learning/breast_cancer_prediction) predicts Breast Cancer based on features derived from images, using SageMaker's Linear Learner.
 - [Ensembling](introduction_to_applying_machine_learning/ensemble_modeling) predicts income using two Amazon SageMaker models to show the advantages in ensembling.
+- [Video Game Sales](introduction_to_applying_machine_learning/video_game_sales) develops a binary prediction model for the success of video games based on review scores.
 
 ### Introduction to Amazon Algorithms
 
@@ -65,8 +66,20 @@ These examples focus on the Amazon SageMaker Python SDK which allows you to writ
 - [TensorFlow and TensorBoard](sagemaker-python-sdk/tensorflow_resnet_cifar10_with_tensorboard)
 - [Distributed TensorFlow](sagemaker-python-sdk/tensorflow_distributed_mnist)
 
+### Under Development
+
+These Amazon SageMaker examples fully illustrate a concept, but may require some additional configuration on the users part to complete.
+
 ## FAQ
 
+*What do I need in order to get started?*
+
+- The quickest setup to run example notebooks includes:
+  - An [AWS account](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-account.html)
+  - Proper [IAM User and Role](http://docs.aws.amazon.com/sagemaker/latest/dg/authentication-and-access-control.html) setup
+  - An [Amazon SageMaker Notebook Instance](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-setup-working-env.html)
+  - An [S3 bucket](http://docs.aws.amazon.com/sagemaker/latest/dg/gs-config-permissions.html)
+
 *Will these examples work outside of Amazon SageMaker Notebook Instances?*
 
 - Although most examples utilize key Amazon SageMaker functionality like distributed, managed training or real-time hosted endpoints, these notebooks can be run outside of Amazon SageMaker Notebook Instances with minimal modification (updating IAM role definition and installing the necessary libraries).

advanced_functionality/README.md

@@ -14,4 +14,4 @@ These examples that showcase unique functionality available in Amazon SageMaker.
 - [Bring Your Own R Algorithm](r_bring_your_own) shows how to bring your own algorithm container to Amazon SageMaker using the R language.
 - [Bring Your Own scikit Algorithm](scikit_bring_your_own) provides a detailed walkthrough on how to package a scikit learn algorithm for training and production-ready hosting.
 - [Bring Your Own MXNet Model](mxnet_mnist_byom) shows how to bring a model trained anywhere using MXNet into Amazon SageMaker
-- [Bring Your Own TensorFlow Model](tensorflow_iris_byom) shows how to bring a model trained anywhere using TensorFlow into Amazon SageMaker
+- [Bring Your Own TensorFlow Model](tensorflow_iris_byom) shows how to bring a model trained anywhere using TensorFlow into Amazon SageMaker

advanced_functionality/data_distribution_types/data_distribution_types.ipynb

@@ -517,7 +517,7 @@
     "sharded_endpoint_config_response = sm.create_endpoint_config(\n",
     "    EndpointConfigName=sharded_endpoint_config,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType': 'ml.c4.2xlarge',\n",
+    "        'InstanceType': 'ml.m4.xlarge',\n",
     "        'InitialInstanceCount': 1,\n",
     "        'ModelName': sharded_job,\n",
     "        'VariantName': 'AllTraffic'}])\n",
@@ -536,7 +536,7 @@
     "replicated_endpoint_config_response = sm.create_endpoint_config(\n",
     "    EndpointConfigName=replicated_endpoint_config,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType': 'ml.c4.2xlarge',\n",
+    "        'InstanceType': 'ml.m4.xlarge',\n",
     "        'InitialInstanceCount': 1,\n",
     "        'ModelName': replicated_job,\n",
     "        'VariantName': 'AllTraffic'}])\n",

advanced_functionality/kmeans_bring_your_own_model/kmeans_bring_your_own_model.ipynb

@@ -236,7 +236,7 @@
     "create_endpoint_config_response = sm.create_endpoint_config(\n",
     "    EndpointConfigName=kmeans_endpoint_config,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType': 'ml.c4.xlarge',\n",
+    "        'InstanceType': 'ml.m4.xlarge',\n",
     "        'InitialInstanceCount': 1,\n",
     "        'ModelName': kmeans_model,\n",
     "        'VariantName': 'AllTraffic'}])\n",

advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb

@@ -433,7 +433,7 @@
    "outputs": [],
    "source": [
     "from sagemaker.predictor import csv_serializer\n",
-    "predictor = tree.deploy(1, 'ml.c4.xlarge', serializer=csv_serializer)"
+    "predictor = tree.deploy(1, 'ml.m4.xlarge', serializer=csv_serializer)"
    ]
   },
   {

advanced_functionality/tensorflow_iris_byom/tensorflow_BYOM_iris.ipynb

@@ -243,7 +243,7 @@
    "source": [
     "### Create endpoint\n",
     "\n",
-    "Now the model is ready to be deployed at a SageMaker endpoint. We can use the ``sagemaker.tensotflow.model.TensorFlowModel.deploy`` method to do this. Unless you have created or prefer other instances, we recommend using 1 ``'ml.m4.xlarge'`` instance for this example. These are supplied as arguments. "
+    "Now the model is ready to be deployed at a SageMaker endpoint. We can use the ``sagemaker.tensorflow.model.TensorFlowModel.deploy`` method to do this. Unless you have created or prefer other instances, we recommend using 1 ``'ml.m4.xlarge'`` instance for this example. These are supplied as arguments. "
    ]
   },
   {

introduction_to_amazon_algorithms/factorization_machines_mnist/factorization_machines_mnist.ipynb

@@ -232,8 +232,7 @@
     "- `feature_dim` is set to 784, which is the number of pixels in each 28 x 28 image.\n",
     "- `predictor_type` is set to 'binary_classifier' since we are trying to predict whether the image is or is not a 0.\n",
     "- `mini_batch_size` is set to 200.  This value can be tuned for relatively minor improvements in fit and speed, but selecting a reasonable value relative to the dataset is appropriate in most cases.\n",
-    "- `num_factors` is set to 10.  As mentioned initially, factorization machines find a lower dimensional representation of the interactions for all features.  Making this value smaller provides a more parsimonious model, closer to a linear model, but may sacrifice information about interactions.  Making it larger provides a higher-dimensional representation of feature interactions, but adds computational complexity and can lead to overfitting.  In a practical application, time should be invested to tune this parameter to the appropriate value.\n",
-    "- `_sparse_input` is set to 'false'.  As mentioned previously, factorization machines are frequently used with sparse data, which is therefore what the algorithm expects.  Setting this to false forces the algorithm to handle the dense recordIO-wrapped protobuf that we created above."
+    "- `num_factors` is set to 10.  As mentioned initially, factorization machines find a lower dimensional representation of the interactions for all features.  Making this value smaller provides a more parsimonious model, closer to a linear model, but may sacrifice information about interactions.  Making it larger provides a higher-dimensional representation of feature interactions, but adds computational complexity and can lead to overfitting.  In a practical application, time should be invested to tune this parameter to the appropriate value."
    ]
   },
   {
@@ -256,8 +255,7 @@
     "fm.set_hyperparameters(feature_dim=784,\n",
     "                      predictor_type='binary_classifier',\n",
     "                      mini_batch_size=200,\n",
-    "                      num_factors=10,\n",
-    "                      _sparse_input='false')\n",
+    "                      num_factors=10)\n",
     "\n",
     "fm.fit({'train': s3_train_data})"
    ]
@@ -279,7 +277,7 @@
    "outputs": [],
    "source": [
     "fm_predictor = fm.deploy(initial_instance_count=1,\n",
-    "                         instance_type='ml.c4.xlarge')"
+    "                         instance_type='ml.m4.xlarge')"
    ]
   },
   {

introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining.ipynb

@@ -393,7 +393,7 @@
     "endpoint_config_response = sage.create_endpoint_config(\n",
     "    EndpointConfigName = endpoint_config_name,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType':'ml.p2.xlarge',\n",
+    "        'InstanceType':'ml.m4.xlarge',\n",
     "        'InitialInstanceCount':1,\n",
     "        'ModelName':model_name,\n",
     "        'VariantName':'AllTraffic'}])\n",

introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-transfer-learning.ipynb

@@ -409,7 +409,7 @@
     "endpoint_config_response = sage.create_endpoint_config(\n",
     "    EndpointConfigName = endpoint_config_name,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType':'ml.p2.xlarge',\n",
+    "        'InstanceType':'ml.m4.xlarge',\n",
     "        'InitialInstanceCount':1,\n",
     "        'ModelName':model_name,\n",
     "        'VariantName':'AllTraffic'}])\n",

introduction_to_amazon_algorithms/lda_topic_modeling/LDA-Introduction.ipynb

@@ -396,7 +396,7 @@
    "source": [
     "lda_inference = lda.deploy(\n",
     "    initial_instance_count=1,\n",
-    "    instance_type='ml.c4.xlarge',  # LDA inference works best on ml.c4 instances\n",
+    "    instance_type='ml.m4.xlarge',  # LDA inference may work better at scale on ml.c4 instances\n",
     ")"
    ]
   },

introduction_to_amazon_algorithms/linear_learner_mnist/linear_learner_mnist.ipynb

@@ -273,7 +273,7 @@
    "outputs": [],
    "source": [
     "linear_predictor = linear.deploy(initial_instance_count=1,\n",
-    "                                 instance_type='ml.c4.xlarge')"
+    "                                 instance_type='ml.m4.xlarge')"
    ]
   },
   {

introduction_to_amazon_algorithms/ntm_synthetic/ntm_synthetic.ipynb

@@ -293,7 +293,7 @@
    "outputs": [],
    "source": [
     "ntm_predictor = ntm.deploy(initial_instance_count=1,\n",
-    "                           instance_type='ml.c4.xlarge')"
+    "                           instance_type='ml.m4.xlarge')"
    ]
   },
   {

introduction_to_amazon_algorithms/pca_mnist/pca_mnist.ipynb

@@ -279,7 +279,7 @@
    "outputs": [],
    "source": [
     "pca_predictor = pca.deploy(initial_instance_count=1,\n",
-    "                           instance_type='ml.c4.xlarge')"
+    "                           instance_type='ml.m4.xlarge')"
    ]
   },
   {

introduction_to_amazon_algorithms/seq2seq_translation_en-de/SageMaker-Seq2Seq-Translation-English-German.ipynb

@@ -461,7 +461,7 @@
     "### Create endpoint configuration\n",
     "Use the model to create an endpoint configuration. The endpoint configuration also contains information about the type and number of EC2 instances to use when hosting the model.\n",
     "\n",
-    "Since SageMaker Seq2Seq is based on Neural Nets, we will use an ml.p2.xlarge (GPU) instance in this example."
+    "Since SageMaker Seq2Seq is based on Neural Nets, we could use an ml.p2.xlarge (GPU) instance, but for this example we will use a free tier eligible ml.m4.xlarge."
    ]
   },
   {
@@ -477,7 +477,7 @@
     "create_endpoint_config_response = sage.create_endpoint_config(\n",
     "    EndpointConfigName = endpoint_config_name,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType':'ml.p2.xlarge',\n",
+    "        'InstanceType':'ml.m4.xlarge',\n",
     "        'InitialInstanceCount':1,\n",
     "        'ModelName':model_name,\n",
     "        'VariantName':'AllTraffic'}])\n",

introduction_to_amazon_algorithms/xgboost_mnist/xgboost_mnist.ipynb

@@ -464,7 +464,7 @@
     "create_endpoint_config_response = sm.create_endpoint_config(\n",
     "    EndpointConfigName = endpoint_config_name,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType':'ml.c4.xlarge',\n",
+    "        'InstanceType':'ml.m4.xlarge',\n",
     "        'InitialVariantWeight':1,\n",
     "        'InitialInstanceCount':1,\n",
     "        'ModelName':model_name,\n",

introduction_to_applying_machine_learning/README.md

@@ -9,3 +9,4 @@ These examples provide a gentle introduction to machine learning concepts as the
 - [Time-series Forecasting](linear_time_series_forecast) generates a forecast for topline product demand using Amazon SageMaker's Linear Learner algorithm.
 - [Cancer Prediction](breast_cancer_prediction) predicts Breast Cancer based on features derived from images, using SageMaker's Linear Learner.
 - [Ensembling](ensemble_modeling) predicts income using two Amazon SageMaker models to show the advantages in ensembling.
+- [Video Game Sales](video_game_sales) develops a binary prediction model for the success of video games based on review scores.

introduction_to_applying_machine_learning/breast_cancer_prediction/Breast Cancer Prediction.ipynb

@@ -472,7 +472,7 @@
     "create_endpoint_config_response = sm.create_endpoint_config(\n",
     "    EndpointConfigName=linear_endpoint_config,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType': 'ml.c4.2xlarge',\n",
+    "        'InstanceType': 'ml.m4.xlarge',\n",
     "        'InitialInstanceCount': 1,\n",
     "        'ModelName': linear_job,\n",
     "        'VariantName': 'AllTraffic'}])\n",
@@ -536,7 +536,7 @@
     "- Total Classification Accuracy \n",
     "- Mean Absolute Error\n",
     "\n",
-    "For our example, we'll keep things simple and use total clssification accuracy as our metric of choice. We will also evalute  Mean Absolute  Error (MAE) as the linear-learner has been optimized using this metric, not necessarily because it is a relevant metric from an application point of view. We'll compare the performance of the linear-learner against a naive benchmark prediction which uses majority class observed in the training data set for prediction on the test data.\n",
+    "For our example, we'll keep things simple and use total classification accuracy as our metric of choice. We will also evaluate  Mean Absolute  Error (MAE) as the linear-learner has been optimized using this metric, not necessarily because it is a relevant metric from an application point of view. We'll compare the performance of the linear-learner against a naive benchmark prediction which uses majority class observed in the training data set for prediction on the test data.\n",
     "\n",
     "\n"
    ]

introduction_to_applying_machine_learning/ensemble_modeling/EnsembleLearnerCensusIncome.ipynb

@@ -942,7 +942,7 @@
     "create_endpoint_config_response = sm.create_endpoint_config(\n",
     "    EndpointConfigName=linear_endpoint_config,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType': 'ml.c4.2xlarge',\n",
+    "        'InstanceType': 'ml.m4.xlarge',\n",
     "        'InitialInstanceCount': 1,\n",
     "        'ModelName': linear_job,\n",
     "        'VariantName': 'AllTraffic'}])\n",

introduction_to_applying_machine_learning/linear_time_series_forecast/linear_time_series_forecast.ipynb

@@ -31,7 +31,7 @@
     "\n",
     "Forecasting is potentially the most broadly relevant machine learning topic there is.  Whether predicting future sales in retail, housing prices in real estate, traffic in cities, or patient visits in healthcare, almost every industry could benefit from improvements in their forecasts.  There are numerous statistical methodologies that have been developed to forecast time-series data, but still, the process for developing forecasts tends to be a mix of objective statistics and subjective interpretations.\n",
     "\n",
-    "Properly modeling time-series data takes a great deal of care.  What's the right level of aggregation to model at?  Too granular and the signal gets lost in the noise, too aggregate and importent variation is missed.  Also, what is the right cyclicality?  Daily, weekly, monthly?  Are there holiday peaks?  How should we weight recent versus overall trends?\n",
+    "Properly modeling time-series data takes a great deal of care.  What's the right level of aggregation to model at?  Too granular and the signal gets lost in the noise, too aggregate and important variation is missed.  Also, what is the right cyclicality?  Daily, weekly, monthly?  Are there holiday peaks?  How should we weight recent versus overall trends?\n",
     "\n",
     "Linear regression with appropriate controls for trend, seasonality, and recent behavior, remains a common method for forecasting stable time-series with reasonable volatility.  This notebook will build a linear model to forecast weekly output for US gasoline products starting in 1991 to 2005.  It will focus almost exclusively on the application.  For a more in-depth treatment on forecasting in general, see [Forecasting: Principles & Practice](https://robjhyndman.com/uwafiles/fpp-notes.pdf).\n",
     "\n",
@@ -374,7 +374,7 @@
    "outputs": [],
    "source": [
     "linear_predictor = linear.deploy(initial_instance_count=1,\n",
-    "                                 instance_type='ml.c4.xlarge')"
+    "                                 instance_type='ml.m4.xlarge')"
    ]
   },
   {

introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb

@@ -243,7 +243,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We see several features that essentially have 100% correlation with one another.  Including these feature pairs in some machine learning algorithms can create catastrophic problems, while in others it will only introduce minor redundancy and bias.  Let's remove them anyway."
+    "We see several features that essentially have 100% correlation with one another.  Including these feature pairs in some machine learning algorithms can create catastrophic problems, while in others it will only introduce minor redundancy and bias.  Let's remove one feature from each of the highly correlated pairs: Day Charge from the pair with Day Mins, Night Charge from the pair with Night Mins, Intl Charge from the pair with Intl Mins:"
    ]
   },
   {
@@ -397,7 +397,6 @@
     "                        subsample=0.8,\n",
     "                        silent=0,\n",
     "                        objective='binary:logistic',\n",
-    "                        num_class=1, \n",
     "                        num_round=100)\n",
     "\n",
     "xgb.fit({'train': s3_input_train, 'validation': s3_input_validation}) "
@@ -420,7 +419,7 @@
    "outputs": [],
    "source": [
     "xgb_predictor = xgb.deploy(initial_instance_count=1,\n",
-    "                           instance_type='ml.c4.xlarge')"
+    "                           instance_type='ml.m4.xlarge')"
    ]
   },
   {

introduction_to_applying_machine_learning/xgboost_direct_marketing/xgboost_direct_marketing_sagemaker.ipynb

@@ -460,7 +460,6 @@
     "                        subsample=0.8,\n",
     "                        silent=0,\n",
     "                        objective='binary:logistic',\n",
-    "                        num_class=1, \n",
     "                        num_round=100)\n",
     "\n",
     "xgb.fit({'train': s3_input_train, 'validation': s3_input_validation}) "
@@ -483,7 +482,7 @@
    "outputs": [],
    "source": [
     "xgb_predictor = xgb.deploy(initial_instance_count=1,\n",
-    "                           instance_type='ml.c4.xlarge')"
+    "                           instance_type='ml.m4.xlarge')"
    ]
   },
   {

sagemaker-python-sdk/1P_kmeans_highlevel/kmeans_mnist.ipynb

@@ -176,7 +176,7 @@
     "%%time\n",
     "\n",
     "kmeans_predictor = kmeans.deploy(initial_instance_count=1,\n",
-    "                                 instance_type='ml.c4.xlarge')"
+    "                                 instance_type='ml.m4.xlarge')"
    ]
   },
   {

sagemaker-python-sdk/1P_kmeans_lowlevel/kmeans_mnist_lowlevel.ipynb

@@ -307,8 +307,8 @@
     "create_endpoint_config_response = sagemaker.create_endpoint_config(\n",
     "    EndpointConfigName = endpoint_config_name,\n",
     "    ProductionVariants=[{\n",
-    "        'InstanceType':'ml.c4.xlarge',\n",
-    "        'InitialInstanceCount':3,\n",
+    "        'InstanceType':'ml.m4.xlarge',\n",
+    "        'InitialInstanceCount':1,\n",
     "        'ModelName':model_name,\n",
     "        'VariantName':'AllTraffic'}])\n",
     "\n",

sagemaker-python-sdk/mxnet_gluon_cifar10/cifar10.ipynb

@@ -141,7 +141,7 @@
    "source": [
     "## Prediction\n",
     "\n",
-    "After training, we use the MXNet estimator object to create and deploy a hosted prediction endpoint. We can use a CPU-based instance for inference (in this case an `ml.c4.xlarge`), even though we trained on GPU instances.\n",
+    "After training, we use the MXNet estimator object to create and deploy a hosted prediction endpoint. We can use a CPU-based instance for inference (in this case an `ml.m4.xlarge`), even though we trained on GPU instances.\n",
     "\n",
     "The predictor object returned by `deploy` lets us call the new endpoint and perform inference on our sample images. "
    ]
@@ -152,7 +152,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "predictor = m.deploy(initial_instance_count=1, instance_type='ml.c4.xlarge')"
+    "predictor = m.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
    ]
   },
   {

sagemaker-python-sdk/mxnet_gluon_mnist/mnist_with_gluon.ipynb

@@ -144,7 +144,7 @@
    },
    "outputs": [],
    "source": [
-    "predictor = m.deploy(initial_instance_count=1, instance_type='ml.c4.xlarge')"
+    "predictor = m.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
    ]
   },
   {