Update Kmeans high and low level examples to use new algorithm image + new Python SDK protobuf conversion (aws#35)

Author: winstonaws

sagemaker-python-sdk/1P_kmeans_highlevel/kmeans_mnist.ipynb

@@ -62,6 +62,19 @@
     "bucket='<bucket-name>'"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "output_location = 's3://{}/kmeans_highlevel_example/output'.format(bucket)\n",
+    "data_location = 's3://{}/kmeans_highlevel_example/data'.format(bucket)\n",
+    "\n",
+    "print('training data will be uploaded to: {}'.format(data_location))\n",
+    "print('training artifacts will be uploaded to: {}'.format(output_location))"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -117,69 +130,13 @@
     "show_digit(train_set[0][30], 'This is a {}'.format(train_set[1][30]))"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Data conversion\n",
-    "\n",
-    "Since algorithms have particular input and output requirements, converting the dataset is also part of the process that a data scientist goes through prior to initiating training. In this particular case, the hosted implementation of k-means takes recordio-wrapped protobuf, where the data we have today is a pickle-ized numpy array on disk.\n",
-    "\n",
-    "Some of the effort involved in the protobuf format conversion is hidden in a library that is imported, below. This library will be folded into the SDK for algorithm authors to make it easier for algorithm authors to support multiple formats. This doesn't __prevent__ algorithm authors from requiring non-standard formats, but it encourages them to support the standard ones.\n",
-    "\n",
-    "For this dataset, conversion takes approximately one minute."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%time\n",
-    "import io\n",
-    "import sagemaker.kmeans\n",
-    "\n",
-    "vectors = [t.tolist() for t in train_set[0]]\n",
-    "labels = [t.tolist() for t in train_set[1]]\n",
-    "\n",
-    "buf = io.BytesIO()\n",
-    "sagemaker.kmeans.write_data_as_pb_recordio(vectors, labels, buf)\n",
-    "buf.seek(0)"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Upload training data"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "%%time\n",
-    "import boto3\n",
-    "\n",
-    "key = 'MNIST-1P-Test/recordio-pb-data'\n",
-    "boto3.resource('s3').Bucket(bucket).Object(key).upload_fileobj(buf)\n",
-    "s3_train_data = 's3://{}/{}'.format(bucket, key)\n",
-    "print('uploaded training data location: {}'.format(s3_train_data))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "output_location = 's3://{}/kmeansoutput'.format(bucket)\n",
-    "print('training artifacts will be uploaded to: {}'.format(output_location))"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -197,14 +154,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from sagemaker.kmeans import KMeans\n",
+    "from sagemaker import KMeans\n",
     "\n",
     "kmeans = KMeans(role=role,\n",
     "                train_instance_count=2,\n",
     "                train_instance_type='ml.c4.8xlarge',\n",
     "                output_path=output_location,\n",
     "                k=10,\n",
-    "                feature_dim=784)"
+    "                data_location=data_location)"
    ]
   },
   {
@@ -215,7 +172,7 @@
    "source": [
     "%%time\n",
     "\n",
-    "kmeans.fit({'train': s3_train_data})"
+    "kmeans.fit(kmeans.record_set(train_set[0]))"
    ]
   },
   {
@@ -279,7 +236,7 @@
     "%%time \n",
     "\n",
     "result = kmeans_predictor.predict(valid_set[0][0:100])\n",
-    "clusters = result['labels']"
+    "clusters = [r.label['closest_cluster'].float32_tensor.values[0] for r in result]"
    ]
   },
   {
@@ -349,7 +306,6 @@
   }
  ],
  "metadata": {
-  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.",
   "kernelspec": {
    "display_name": "Environment (conda_python3)",
    "language": "python",
@@ -366,7 +322,8 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.6.3"
-  }
+  },
+  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2