Merge pull request aws#63 from awslabs/arpin_bash_organize

Updated: READMEs and organization

Merging in preparation for Monday's meeting

Author: djarpin

README.md

@@ -17,7 +17,7 @@ These examples provide a gentle introduction to machine learning concepts as the
 
 These examples provide quick walkthroughs to get you up and running with Amazon SageMaker's custom developed algorithms.  Most of these algorithms can train on distributed hardware, scale incredibly well, and are faster and cheaper than popular alternatives.
 
-- [k-means](introduction_to_amazon_algorithms/1P_kmeans_highlevel) is our introductory example for Amazon SageMaker.  It walks through the process of clustering MNIST images of handwritten digits using Amazon SageMaker k-means.
+- [k-means](sagemaker-python-sdk/1P_kmeans_highlevel) is our introductory example for Amazon SageMaker.  It walks through the process of clustering MNIST images of handwritten digits using Amazon SageMaker k-means.
 - [Factorization Machines](introduction_to_amazon_algorithms/factorization_machines_mnist) showcases Amazon SageMaker's implementation of the algorithm to predict whether a handwritten digit from the MNIST dataset is a 0 or not using a binary classifier.
 - [Latent Dirichlet Allocation (LDA)](introduction_to_amazon_algorithms/lda_topic_modeling) introduces topic modeling using Amazon SageMaker Latent Dirichlet Allocation (LDA) on a synthetic dataset.
 - [Linear Learner](introduction_to_amazon_algorithms/linear_learner_mnist) predicts whether a handwritten digit from the MNIST dataset is a 0 or not using a binary classifier from Amazon SageMaker Linear Learner.
@@ -35,19 +35,44 @@ These examples provide more thorough mathematical treatment on a select group of
 
 ### Advanced Amazon SageMaker Functionality
 
+These examples that showcase unique functionality available in Amazon SageMaker.  They cover a broad range of topics and will utilize a variety of methods, but aim to provide the user with sufficient insight or inspiration to develop within Amazon SageMaker.
+
+- [Data Distribution Types](advanced_functionality/data_distribution_types) showcases the difference between two methods for sending data from S3 to Amazon SageMaker Training instances.  This has particular implication for scalability and accuracy of distributed training.
+- [Encrypting Your Data](advanced_functionality/handling_kms_encrypted_data) shows how to use Server Side KMS encrypted data with Amazon SageMaker training works. The IAM role used for S3 access needs to have permissions to encrypt and decrypt data with the KMS key.
+- [Using Parquet Data](advanced_functionality/parquet_to_recordio_protobuf) shows how to bring [Parquet](https://parquet.apache.org/) data sitting in S3 into an Amazon SageMaker Notebook and convert it into the recordIO-protobuf format that many SageMaker algorithms consume. 
+- [Connecting to Redshift](advanced_functionality/working_with_redshift_data) demonstrates how to copy data from Redshift to S3 and vice-versa without leaving Amazon SageMaker Notebooks.
+- [Bring Your Own XGBoost Model](advanced_functionality/xgboost_bring_your_own_model) shows how to use Amazon SageMaker Algorithms containers to bring a pre-trained model to a realtime hosted endpoint without ever needing to think about REST APIs.
+- [Bring Your Own k-means Model](advanced_functionality/kmeans_bring_your_own_model) shows how to take a model that's been fit elsewhere and use Amazon SageMaker Algorithms containers to host it.
 - [Installing the R Kernel](advanced_functionality/install_r_kernel) shows how to install the R kernel into an Amazon SageMaker Notebook Instance.
-- [Bring Your Own Model for k-means](advanced_functionality/kmeans_bring_your_own_model) shows how to take a model that's been fit elsewhere and use Amazon SageMaker Algorithms containers to host it.
-- [Bring Your Own Algorithm with R](advanced_functionality/r_bring_your_own) shows how to bring your own algorithm container to Amazon SageMaker using the R language.
-- [Bring Your Own Tensorflow Model](sagemaker-python-sdk/tensorflow_iris_byom) shows how to bring a model trained anywhere into Amazon SageMaker
-- [Bring Your Own MXNet Model](sagemaker-python-sdk/tensorflow_iris_byom) shows how to bring a model trained anywhere using MXNet into Amazon SageMaker
-- [Bring Your Own TensorFlow Model](sagemaker-python-sdk/tensorflow_iris_byom) shows how to bring a model trained anywhere using TensorFlow into Amazon SageMaker
+- [Bring Your Own R Algorithm](advanced_functionality/r_bring_your_own) shows how to bring your own algorithm container to Amazon SageMaker using the R language.
+- [Bring Your Own scikit Algorithm](advanced_functionality/scikit_bring_your_own) provides a detailed walkthrough on how to package a scikit learn algorithm for training and production-ready hosting.
+
+### Amazon SageMaker TensorFlow and MXNet Pre-Built Containers and the Python SDDK
+
+These examples focus on the Amazon SageMaker Python SDK which allows you to write idiomatic TensorFlow or MXNet and then train or host in pre-built containers.
+
+- [cifar 10 with MXNet Gluon](sagemaker-python-sdk/mxnet_gluon_cifar10)
+- [MNIST with MXNet Gluon](sagemaker-python-sdk/mxnet_gluon_mnist)
+- [MNIST with MXNet](sagemaker-python-sdk/mxnet_mnist)
+- [TensorFlow Neural Networks with Layers](sagemaker-python-sdk/tensorflow_abalone_age_predictor_using_layers)
+- [TensorFlow Networks with Keras](sagemaker-python-sdk/tensorflow_abalone_age_predictor_keras)
+- [Introduction to Estimators in TensorFlow](sagemaker-python-sdk/tensorflow_iris_dnn_classifier_using_estimators)
+- [TensorFlow and TensorBoard](sagemaker-python-sdk/tensorflow_resnet_cifar10_with_tensorboard)
+
+### Under Development
+
+These Amazon SageMaker examples fully illustrate a concept, but may require some additional configuration on the users part to complete.
+
+- [Bring Your Own MXNet Model](under_development/tensorflow_iris_byom) shows how to bring a model trained anywhere using MXNet into Amazon SageMaker
+- [Bring Your Own TensorFlow Model](under_development/tensorflow_iris_byom) shows how to bring a model trained anywhere using TensorFlow into Amazon SageMaker
+- [Ensembling Multiple Models](under_development/modeling) creates two different models for prediction, hosts them independently and shows how their outputs can be combined for better accuracy than either one alone.
 
 ## FAQ
 
-*Will these examples work outside of Amazon SageMaker?*
+*Will these examples work outside of Amazon SageMaker Notebook Instances?*
 
 - Although most examples utilize key Amazon SageMaker functionality like distributed, managed training or real-time hosted endpoints, these notebooks can be run outside of Amazon SageMaker Notebook Instances with minimal modification (updating IAM role definition and installing the necessary libraries).
 
 *How do I contribute my own example notebook?*
 
-- Although we're extremely excited to receive contributions from the community, we're still working on the best mechanism to take in examples from and external source.  Please bear with us in the short-term if pull requests take longer than expected or are closed.
+- Although we're extremely excited to receive contributions from the community, we're still working on the best mechanism to take in examples from and external source.  Please bear with us in the short-term if pull requests take longer than expected or are closed.

advanced_functionality/README.md

@@ -1,13 +1,15 @@
-# Advanced Functionality
+# Amazon SageMaker Examples
 
-This directory includes examples which showcase unique functionality available in Amazon SageMaker.  Examples cover a broad range of topics and will utilize a variety of methods, but aim to provide the user with sufficient insight or inspiration to develop within Amazon SageMaker.
+### Advanced Amazon SageMaker Functionality
 
-Example Notebooks include:
-- *data_distribution_types*: Showcases the difference between two methods for sending data from S3 to Amazon SageMaker Training instances.  This has particular implication for scalability and accuracy of distributed training.
-- *install_r_kernel*: A quick introduction to getting R installed and running within Amazon SageMaker Notebook Instances.
-- *kmeans_bring_your_own_model*: How to use Amazon SageMaker Algorithms containers to bring a pre-trained model to a realtime hosted endpoint without ever needing to think about REST APIs.
-- *r_bring_your_own*: How to containerize an R algorithm using Docker and plumber for hosting so that it can be used in Amazon SageMaker's managed training and realtime hosting.
-- *xgboost_bring_your_own_model*: How to use Amazon SageMaker Algorithms containers to bring a pre-trained model to a realtime hosted endpoint without ever needing to think about REST APIs.
-- *handling_kms_encrypted_data.ipynb*: How to use Server Side KMS encrypted data with Amazon SageMaker training works. The IAM role used for S3 access needs to have permissions to encrypt and decrypt data with the KMS key.
-- *parquet_to_recordio_protobuf.ipynb*: How to convert Parquet data format into the recordIO-protobuf format that many SageMaker algorithms consume. 
-- *working_with_redshift_data.ipynb*: Demonstrates how to copy data from Redshift to S3 and vice-versa. 
+These examples that showcase unique functionality available in Amazon SageMaker.  They cover a broad range of topics and will utilize a variety of methods, but aim to provide the user with sufficient insight or inspiration to develop within Amazon SageMaker.
+
+- [Data Distribution Types](data_distribution_types) showcases the difference between two methods for sending data from S3 to Amazon SageMaker Training instances.  This has particular implication for scalability and accuracy of distributed training.
+- [Encrypting Your Data](handling_kms_encrypted_data) shows how to use Server Side KMS encrypted data with Amazon SageMaker training works. The IAM role used for S3 access needs to have permissions to encrypt and decrypt data with the KMS key.
+- [Using Parquet Data](parquet_to_recordio_protobuf) shows how to bring [Parquet](https://parquet.apache.org/) data sitting in S3 into an Amazon SageMaker Notebook and convert it into the recordIO-protobuf format that many SageMaker algorithms consume. 
+- [Connecting to Redshift](working_with_redshift_data) demonstrates how to copy data from Redshift to S3 and vice-versa without leaving Amazon SageMaker Notebooks.
+- [Bring Your Own XGBoost Model](xgboost_bring_your_own_model) shows how to use Amazon SageMaker Algorithms containers to bring a pre-trained model to a realtime hosted endpoint without ever needing to think about REST APIs.
+- [Bring Your Own k-means Model](kmeans_bring_your_own_model) shows how to take a model that's been fit elsewhere and use Amazon SageMaker Algorithms containers to host it.
+- [Installing the R Kernel](install_r_kernel) shows how to install the R kernel into an Amazon SageMaker Notebook Instance.
+- [Bring Your Own R Algorithm](r_bring_your_own) shows how to bring your own algorithm container to Amazon SageMaker using the R language.
+- [Bring Your Own scikit Algorithm](advanced_functionality/scikit_bring_your_own) provides a detailed walkthrough on how to package a scikit learn algorithm for training and production-ready hosting.

introduction_to_amazon_algorithms/README.md

@@ -1,15 +1,15 @@
-# Introduction to Amazon Algorithms
+# Amazon SageMaker Examples
 
-This directory includes introductory examples to Amazon SageMaker Algorithms that we have developed so far.  It seeks to provide guidance and examples on basic functionality rather than a detailed scientific review or an implementation on complex, real-world data.
+### Introduction to Amazon Algorithms
 
-Example Notebooks include:
-- *1P_kmeans_highlevel*: Our introduction to Amazon SageMaker which walks through the process of clustering MNIST images of handwritten digits.
-- *factorization_machines_mnist*: Predicts whether a handwritten digit from the MNIST dataset is a 0 or not using a binary classifier from Amazon SageMaker Factorization Machines.
-- *lda_topic_modeling*: Topic modeling using Amazon SageMaker Latent Dirichlet Allocation (LDA) on a synthetic dataset.
-- *linear_mnist*: Predicts whether a handwritten digit from the MNIST dataset is a 0 or not using a binary classifier from Amazon SageMaker Linear Learner.
-- *ntm_synthetic*: Uses Amazon SageMaker Neural Topic Model (NTM) to uncover topics in documents from a synthetic data source, where topic distributions are known.
-- *pca_mnist*: Uses Amazon SageMaker Principal Components Analysis (PCA) to calculate eigendigits from MNIST.
-- *seq2seq*: Seq2Seq algorithm is built on top of [Sockeye](https://github.com/awslabs/sockeye), a sequence-to-sequence framework for Neural Machine Translation based on MXNet. SageMaker Seq2Seq implements state-of-the-art encoder-decoder architectures which can also be used for tasks like Abstractive Summarization in addition to Machine Translation.
-- *xgboost_abalone*: Predicts the age of abalone ([Abalone dataset](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html)) using regression from Amazon SageMaker XGBoost.
-- *xgboost_mnist*: Uses Amazon SageMaker XGBoost to classifiy handwritten digits from the MNIST dataset into one of the ten digits using a multi-class classifier. Both single machine and distributed use-cases are presented.
-- *image_classification*: Uses Amazon SageMaker Image classification algorithm to train a Resnet on [caltech dataset](http://www.vision.caltech.edu/Image_Datasets/Caltech256/), either from scratch or using a pre-trained model.
+These examples provide quick walkthroughs to get you up and running with Amazon SageMaker's custom developed algorithms.  Most of these algorithms can train on distributed hardware, scale incredibly well, and are faster and cheaper than popular alternatives.
+
+- [k-means](../sagemaker-python-sdk/1P_kmeans_highlevel) is our introductory example for Amazon SageMaker.  It walks through the process of clustering MNIST images of handwritten digits using Amazon SageMaker k-means.
+- [Factorization Machines](factorization_machines_mnist) showcases Amazon SageMaker's implementation of the algorithm to predict whether a handwritten digit from the MNIST dataset is a 0 or not using a binary classifier.
+- [Latent Dirichlet Allocation (LDA)](lda_topic_modeling) introduces topic modeling using Amazon SageMaker Latent Dirichlet Allocation (LDA) on a synthetic dataset.
+- [Linear Learner](linear_learner_mnist) predicts whether a handwritten digit from the MNIST dataset is a 0 or not using a binary classifier from Amazon SageMaker Linear Learner.
+- [Neural Topic Model (NTM)](ntm_synthetic) uses Amazon SageMaker Neural Topic Model (NTM) to uncover topics in documents from a synthetic data source, where topic distributions are known.
+- [Principal Components Analysis (PCA)](pca_mnist) uses Amazon SageMaker PCA to calculate eigendigits from MNIST.
+- [Seq2Seq](seq2seq) uses the Amazon SageMaker Seq2Seq algorithm that's built on top of [Sockeye](https://github.com/awslabs/sockeye), which is a sequence-to-sequence framework for Neural Machine Translation based on MXNet.  Seq2Seq implements state-of-the-art encoder-decoder architectures which can also be used for tasks like Abstractive Summarization in addition to Machine Translation.  This notebook shows translation from English to German text.
+- [XGBoost for regression](xgboost_abalone) predicts the age of abalone ([Abalone dataset](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression.html)) using regression from Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost).
+- [XGBoost for multi-class classification](xgboost_mnist) uses Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost) to classifiy handwritten digits from the MNIST dataset as one of the ten digits using a multi-class classifier. Both single machine and distributed use-cases are presented.

introduction_to_applying_machine_learning/README.md

@@ -1,8 +1,10 @@
-# Introduction to Applying Machine Learning
+# Amazon SageMaker Examples
 
-This directory includes introductory examples for applying machine learning to real world problems using Amazon SageMaker.  The notebooks tend to provide descriptive overviews rather than dense mathematical content to provide an easy on-ramp to using machine learning in common problems faced across a variety of industries.
+### Introduction to Applying Machine Learning
 
-Example Notebooks include:
-- *linear_time_series_forecast*: Generate a time-series forecast for product demand using Amazon SageMaker's Linear Learner algorithm.
-- *xgboost_customer_churn*: Predict customer churn by building a highly predictive gradient boosted trees model using Amazon SageMaker's implementation of the popular open source XGBoost package.
-- *xgboost_direct_marketing*: Predict successful conversion of prospective customers using gradient boosted trees from Amazon SageMaker's implementation of the popular open source XGBoost package.
+These examples provide a gentle introduction to machine learning concepts as they are applied in practical use cases across a variety of sectors.
+
+- [Targeted Direct Marketing](xgboost_direct_marketing) predicts potential customers that are most likely to convert based on customer and aggregate level metrics, using Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost).
+- [Predicting Customer Churn](xgboost_customer_churn) uses customer interaction and service usage data to find those most likely to churn, and then walks through the cost/benefit trade-offs of providing retention incentives.  This uses Amazon SageMaker's implementation of [XGBoost](https://github.com/dmlc/xgboost) to create a highly predictive model.
+- [Time-series Forecasting](linear_time_series_forecast) generates a forecast for topline product demand using Amazon SageMaker's Linear Learner algorithm.
+- [Cancer Prediction](breast_cancer_prediction) predicts Breast Cancer based on features derived from images, using SageMaker's Linear Learner.

sagemaker-python-sdk/README.md

@@ -0,0 +1,13 @@
+# Amazon SageMaker Examples
+
+### Amazon SageMaker TensorFlow and MXNet Pre-Built Containers and the Python SDDK
+
+These examples focus on the Amazon SageMaker Python SDK which allows you to write idiomatic TensorFlow or MXNet and then train or host in pre-built containers.
+
+- [cifar 10 with MXNet Gluon](mxnet_gluon_cifar10)
+- [MNIST with MXNet Gluon](mxnet_gluon_mnist)
+- [MNIST with MXNet](mxnet_mnist)
+- [TensorFlow Neural Networks with Layers](tensorflow_abalone_age_predictor_using_layers)
+- [TensorFlow Networks with Keras](tensorflow_abalone_age_predictor_keras)
+- [Introduction to Estimators in TensorFlow](tensorflow_iris_dnn_classifier_using_estimators)
+- [TensorFlow and TensorBoard](tensorflow_resnet_cifar10_with_tensorboard)

scientific_details_of_algorithms/README.md

@@ -0,0 +1,7 @@
+# Amazon SageMaker Examples
+
+### Scientific Details of Algorithms
+
+These examples provide more thorough mathematical treatment on a select group of algorithms.
+
+- [Latent Dirichlet Allocation (LDA)](lda_topic_modeling) dives into Amazon SageMaker's spectral decomposition approach to LDA.