Merge pull request aws#43 from awslabs/arpin_advanced_functionality

Updated: advanced_functionality notebooks

Pushing to prepare for Monday's meeting.

Author: djarpin

advanced_functionality/data_distribution_types/AmazonAIAlgorithmsIO_pb2.py

@@ -31,7 +31,7 @@
     "---\n",
     "# Setup\n",
     "\n",
-    "_This notebook was created and tested on an ml.m4xlarge notebook instance._\n",
+    "_This notebook was created and tested on an ml.m4.xlarge notebook instance._\n",
     "\n",
     "Let's start by specifying:\n",
     "\n",
@@ -78,11 +78,12 @@
     "import matplotlib.pyplot as plt\n",
     "from IPython.display import display\n",
     "import io\n",
-    "import convert_data\n",
     "import time\n",
     "import copy\n",
     "import json\n",
-    "import sys"
+    "import sys\n",
+    "import sagemaker.amazon.common as smac\n",
+    "import os"
    ]
   },
   {
@@ -160,7 +161,7 @@
    "metadata": {},
    "source": [
     "We can see:\n",
-    "- `EventCode` is pretty unevently distributed, with some events making up 7%+ of the observations and others being a thousandth of a percent.\n",
+    "- `EventCode` is pretty unevenly distributed, with some events making up 7%+ of the observations and others being a thousandth of a percent.\n",
     "- `AvgTone` seems to be reasonably smoothly distributed, while `NumArticles` has a long tail, and `Actor` geo features have suspiciously large spikes near 0.\n",
     "\n",
     "Let's remove the (0, 0) lat-longs, one hot encode `EventCode`, and prepare our data for a machine learning model.  For this example we'll keep things straightforward and try to predict `AvgTone`, using the other variables in our dataset as features.\n",
@@ -193,12 +194,10 @@
    "outputs": [],
    "source": [
     "def write_to_s3(bucket, prefix, channel, file_prefix, X, y):\n",
-    "    f = io.BytesIO()\n",
-    "    feature_size = X.shape[1]\n",
-    "    for features, target in zip(X, y):\n",
-    "        convert_data.write_recordio(f, convert_data.list_to_record_bytes(features, label=target, feature_size=feature_size))\n",
-    "    f.seek(0)\n",
-    "    boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, channel, file_prefix + '.data')).upload_fileobj(f)\n",
+    "    buf = io.BytesIO()\n",
+    "    smac.write_numpy_to_dense_tensor(buf, X.astype('float32'), y.astype('float32'))\n",
+    "    buf.seek(0)\n",
+    "    boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, channel, file_prefix + '.data')).upload_fileobj(buf)\n",
     "\n",
     "def transform_gdelt(df, events=None):\n",
     "    df = df[['AvgTone', 'EventCode', 'NumArticles', 'Actor1Geo_Lat', 'Actor1Geo_Long', 'Actor2Geo_Lat', 'Actor2Geo_Long']]\n",
@@ -649,7 +648,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next, because POST requests to our endpoint are limited to ~6MB, we'll setup a small function to split our test data up into mini-batches, loop through and invoke our endpoint to get predictions, and gather them into a single array."
+    "Next, because POST requests to our endpoint are limited to ~6MB, we'll setup a small function to split our test data up into mini-batches that are each about 5MB, loop through and invoke our endpoint to get predictions for those mini-batches, and gather them into a single array."
    ]
   },
   {
@@ -714,10 +713,28 @@
     "\n",
     "Different algorithms can be expected to show variation in which distribution mechanism is most effective at achieving optimal compute spend per point of model accuracy.  The message remains the same though, that the process of finding the right distribution type is another experiment in optimizing model training times."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### (Optional) Clean-up\n",
+    "\n",
+    "If you're ready to be done with this notebook, please uncomment and run the cell below.  This will remove the hosted endpoints you created and avoid any charges from a stray instance being left on."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#sm.delete_endpoint(EndpointName=sharded_endpoint)\n",
+    "#sm.delete_endpoint(EndpointName=replicated_endpoint)"
+   ]
   }
  ],
  "metadata": {
-  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.",
   "kernelspec": {
    "display_name": "Environment (conda_python3)",
    "language": "python",
@@ -734,7 +751,8 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.6.3"
-  }
+  },
+  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2