improve markdown

Author: andremoeller

sagemaker-spark/pyspark_mnist/pyspark_mnist_kmeans.ipynb

@@ -1,34 +1,17 @@
 {
  "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n",
-    "\n",
-    "Licensed under the Apache License, Version 2.0 (the \"License\").\n",
-    "You may not use this file except in compliance with the License.\n",
-    "A copy of the License is located at\n",
-    " \n",
-    "  http://aws.amazon.com/apache2.0/\n",
-    "\n",
-    "or in the \"license\" file accompanying this file. This file is distributed\n",
-    "on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either\n",
-    "express or implied. See the License for the specific language governing\n",
-    "permissions and limitations under the License."
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# SageMaker PySpark K-Means Clustering MNIST Example\n",
     "\n",
     "1. [Introduction](#Introduction)\n",
-    "2. [Loading the Data](#Loading-the-Data)\n",
-    "3. [Training and Hosting a Model](#Training-and-Hosting-a-Model)\n",
-    "4. [Inference](#Inference)\n",
-    "5. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n"
+    "2. [Setup](#Setup)\n",
+    "3. [Loading the Data](#Loading-the-Data)\n",
+    "4. [Training and Hosting a Model](#Training-and-Hosting-a-Model)\n",
+    "5. [Inference](#Inference)\n",
+    "6. [More on SageMaker Spark](#More-on-SageMaker-Spark)\n"
    ]
   },
   {
@@ -38,15 +21,20 @@
     "## Introduction\n",
     "This notebook will show how to classify handwritten digits using the K-Means clustering algorithm through the SageMaker PySpark library. We will train on Amazon SageMaker using K-Means clustering on the MNIST dataset, host the trained model on Amazon SageMaker, and then make predictions against that hosted model.\n",
     "\n",
-    "You can visit SageMaker Spark's Github repository at https://github.com/aws/sagemaker-spark for more about SageMaker Spark.\n",
+    "Unlike the other notebooks that demonstrate K-Means clustering on Amazon SageMaker, this notebook uses a SparkSession to manipulate data, and uses the SageMaker Spark library to interact with SageMaker with Spark Estimators and Transformers.\n",
     "\n",
-    "We will train on Amazon SageMaker using the KMeans Clustering on the MNIST dataset, host the trained model on Amazon SageMaker, and then make predictions against that hosted model.\n",
+    "You can visit SageMaker Spark's GitHub repository at https://github.com/aws/sagemaker-spark to learn more about SageMaker Spark.\n",
     "\n",
-    "First, we load the MNIST dataset into a Spark Dataframe, which dataset is available in LibSVM format at\n",
-    "\n",
-    "`s3://sagemaker-sample-data-[region]/spark/mnist/train/`\n",
+    "This notebook was created and tested on an ml.m4.xlarge notebook instance."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
     "\n",
-    "where `[region]` is replaced with a supported AWS region, such as us-east-1"
+    "First, we import the necessary modules and create the SparkSession with the SageMaker Spark dependencies."
    ]
   },
   {
@@ -63,10 +51,8 @@
     "from pyspark.sql import SparkSession\n",
     "\n",
     "import sagemaker\n",
-    "import sagemaker_pyspark\n",
     "from sagemaker import get_execution_role\n",
-    "\n",
-    "sagemaker_session = sagemaker.Session()\n",
+    "import sagemaker_pyspark\n",
     "\n",
     "role = get_execution_role()\n",
     "\n",
@@ -81,34 +67,23 @@
     "    .master(\"local[*]\").getOrCreate()"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import boto3\n",
-    "\n",
-    "# use the region-specific sample data bucket\n",
-    "region = boto3.Session().region_name\n",
-    "trainingData = spark.read.format('libsvm')\\\n",
-    "    .option('numFeatures', '784')\\\n",
-    "    .load('s3a://sagemaker-sample-data-{}/spark/mnist/train/'.format(region))\n",
-    "\n",
-    "testData = spark.read.format('libsvm')\\\n",
-    "    .option('numFeatures', '784')\\\n",
-    "    .load('s3a://sagemaker-sample-data-{}/spark/mnist/test/'.format(region))"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Loading the Data\n",
     "\n",
+    "Now, we load the MNIST dataset into a Spark Dataframe, which dataset is available in LibSVM format at\n",
+    "\n",
+    "`s3://sagemaker-sample-data-[region]/spark/mnist/train/`\n",
+    "\n",
+    "where `[region]` is replaced with a supported AWS region, such as us-east-1.\n",
+    "\n",
     "In order to train and make inferences our input DataFrame must have a column of Doubles (named \"label\" by default) and a column of Vectors of Doubles (named \"features\" by default).\n",
     "\n",
-    "Spark's LibSVM DataFrameReader loads a DataFrame already suitable for training and inference."
+    "Spark's LibSVM DataFrameReader loads a DataFrame already suitable for training and inference.\n",
+    "\n",
+    "Here, we load into a DataFrame in the SparkSession running on the local Notebook Instance, but you can connect your Notebook Instance to a remote Spark cluster for heavier workloads. Starting from EMR 5.11.0, SageMaker Spark is pre-installed on EMR Spark clusters. For more on connecting your SageMaker Notebook Instance to a remote EMR cluster, please see [this blog post](https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/)."
    ]
   },
   {
@@ -119,6 +94,18 @@
    },
    "outputs": [],
    "source": [
+    "import boto3\n",
+    "\n",
+    "region = boto3.Session().region_name\n",
+    "\n",
+    "trainingData = spark.read.format('libsvm')\\\n",
+    "    .option('numFeatures', '784')\\\n",
+    "    .load('s3a://sagemaker-sample-data-{}/spark/mnist/train/'.format(region))\n",
+    "\n",
+    "testData = spark.read.format('libsvm')\\\n",
+    "    .option('numFeatures', '784')\\\n",
+    "    .load('s3a://sagemaker-sample-data-{}/spark/mnist/test/'.format(region))\n",
+    "\n",
     "trainingData.show()"
    ]
   },
@@ -151,9 +138,9 @@
     "\n",
     "kmeans_estimator = KMeansSageMakerEstimator(\n",
     "    sagemakerRole=IAMRole(role),\n",
-    "    trainingInstanceType='ml.p2.xlarge',\n",
+    "    trainingInstanceType='ml.m4.xlarge',\n",
     "    trainingInstanceCount=1,\n",
-    "    endpointInstanceType='ml.c4.xlarge',\n",
+    "    endpointInstanceType='ml.m4.xlarge',\n",
     "    endpointInitialInstanceCount=1)\n",
     "\n",
     "kmeans_estimator.setK(10)\n",
@@ -170,7 +157,7 @@
     "## Inference\n",
     "\n",
     "Now we transform our DataFrame.\n",
-    "To do this, we serialize each row's \"features\" Vector of Doubles into a Protobuf format for inference against the Amazon SageMaker Endpoint. We deserialize the Protobuf responses back into our DataFrame:"
+    "To do this, we serialize each row's \"features\" Vector of Doubles into a Protobuf format for inference against the Amazon SageMaker Endpoint. We deserialize the Protobuf responses back into our DataFrame. This serialization and deserialization is handled automatically by the `transform()` method:"
    ]
   },
   {
@@ -186,6 +173,13 @@
     "transformedData.show()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "How well did the algorithm perform? Let us display the digits from each of the clusters and manually inspect the results:"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -278,7 +272,8 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.6.2"
-  }
+  },
+  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2