Skip to content

Commit e37174d

Browse files
djarpindjarpin
authored
Merge pull request aws#296 from juliodelgadoaws/master
Use SageMaker SDK RandomCutForest estimator
2 parents 6283c40 + b63c6a1 commit e37174d

File tree

1 file changed

+34
-128
lines changed

1 file changed

+34
-128
lines changed

‎introduction_to_amazon_algorithms/random_cut_forest/random_cut_forest.ipynb

Lines changed: 34 additions & 128 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@
5050
"\n",
5151
"*This notebook was created and tested on an ml.m4.xlarge notebook instance.*\n",
5252
"\n",
53-
"Our first step is to setup our AWS credentials so that AWS SageMaker can store and access data. We also need some data to inspect and to train upon."
53+
"Our first step is to setup our AWS credentials so that AWS SageMaker can store and access training data and model artifacts. We also need some data to inspect and to train upon."
5454
]
5555
},
5656
{
@@ -203,46 +203,6 @@
203203
"taxi_data[5952:6000]"
204204
]
205205
},
206-
{
207-
"cell_type": "markdown",
208-
"metadata": {},
209-
"source": [
210-
"## Store Data on S3\n",
211-
"\n",
212-
"The Random Cut Forest Algorithm accepts data in [RecordIO](https://mxnet.apache.org/api/python/io/io.html#module-mxnet.recordio) [Protobuf](https://developers.google.com/protocol-buffers/) format. The SageMaker Python API provides helper functions for easily converting your data into this format. Below we convert the taxicab data and upload it to the `bucket + prefix` Amazon S3 destination specified at the beginning of this notebook in the [Setup AWS Credentials](#Setup-AWS-Credentials) section."
213-
]
214-
},
215-
{
216-
"cell_type": "code",
217-
"execution_count": null,
218-
"metadata": {},
219-
"outputs": [],
220-
"source": [
221-
"def convert_and_upload_training_data(ndarray, bucket, prefix, filename='data.pbr'):\n",
222-
" import boto3\n",
223-
" import os\n",
224-
" from sagemaker.amazon.common import numpy_to_record_serializer\n",
225-
" \n",
226-
" # convert numpy array to Protobuf RecordIO format\n",
227-
" serializer = numpy_to_record_serializer()\n",
228-
" buffer = serializer(ndarray)\n",
229-
" \n",
230-
" # Upload to S3\n",
231-
" s3_object = os.path.join(prefix, 'train', filename)\n",
232-
" boto3.Session().resource('s3').Bucket(bucket).Object(s3_object).upload_fileobj(buffer)\n",
233-
" \n",
234-
" s3_path = 's3://{}/{}'.format(bucket, s3_object)\n",
235-
" return s3_path\n",
236-
"\n",
237-
"\n",
238-
"# RCV only works on an array of values.\n",
239-
"s3_train_data = convert_and_upload_training_data(\n",
240-
" taxi_data.value.as_matrix().reshape(-1,1),\n",
241-
" bucket,\n",
242-
" prefix)\n",
243-
"print('Uploaded data to {}'.format(s3_train_data))"
244-
]
245-
},
246206
{
247207
"cell_type": "markdown",
248208
"metadata": {},
@@ -251,30 +211,7 @@
251211
"\n",
252212
"***\n",
253213
"\n",
254-
"We have created a training data set and uploaded it to S3. Next, we configure a SageMaker training job to use the Random Cut Forest (RCF) algorithm on said training data.\n",
255-
"\n",
256-
"The first step is to specify the location of the Docker image containing the SageMaker Random Cut Forest algorithm. In order to minimize communication latency, we provide containers for each AWS region in which SageMaker is available. The code below automatically chooses an algorithm container based on the current region; that is, the region in which this notebook is run."
257-
]
258-
},
259-
{
260-
"cell_type": "code",
261-
"execution_count": null,
262-
"metadata": {},
263-
"outputs": [],
264-
"source": [
265-
"import boto3\n",
266-
"\n",
267-
"# select the algorithm container based on this notebook's current location\n",
268-
"containers = {'us-west-2': '174872318107.dkr.ecr.us-west-2.amazonaws.com/randomcutforest:latest',\n",
269-
" 'us-east-1': '382416733822.dkr.ecr.us-east-1.amazonaws.com/randomcutforest:latest',\n",
270-
" 'us-east-2': '404615174143.dkr.ecr.us-east-2.amazonaws.com/randomcutforest:latest',\n",
271-
" 'eu-west-1': '438346466558.dkr.ecr.eu-west-1.amazonaws.com/randomcutforest:latest',\n",
272-
" 'ap-northeast-1': '351501993468.dkr.ecr.ap-northeast-1.amazonaws.com/randomcutforest:latest',\n",
273-
" 'ap-northeast-2': '835164637446.dkr.ecr.ap-northeast-2.amazonaws.com/randomcutforest:latest'}\n",
274-
"region_name = boto3.Session().region_name\n",
275-
"container = containers[region_name]\n",
276-
"\n",
277-
"print('Using SageMaker RCF container: {} ({})'.format(container, region_name))"
214+
"Next, we configure a SageMaker training job to train the Random Cut Forest (RCF) algorithm on the taxi cab data."
278215
]
279216
},
280217
{
@@ -302,35 +239,21 @@
302239
"metadata": {},
303240
"outputs": [],
304241
"source": [
242+
"from sagemaker import RandomCutForest\n",
243+
"\n",
305244
"session = sagemaker.Session()\n",
306245
"\n",
307246
"# specify general training job information\n",
308-
"rcf = sagemaker.estimator.Estimator(\n",
309-
" container,\n",
310-
" execution_role,\n",
311-
" output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
312-
" train_instance_count=1,\n",
313-
" train_instance_type='ml.m4.xlarge',\n",
314-
" sagemaker_session=session,\n",
315-
")\n",
316-
"\n",
317-
"# set algorithm-specific hyperparameters\n",
318-
"rcf.set_hyperparameters(\n",
319-
" num_samples_per_tree = 200,\n",
320-
" num_trees = 50,\n",
321-
" feature_dim = 1,\n",
322-
")\n",
323-
"\n",
324-
"# RCF training requires sharded data. See documentation for\n",
325-
"# more information.\n",
326-
"s3_train_input = sagemaker.session.s3_input(\n",
327-
" s3_train_data,\n",
328-
" distribution='ShardedByS3Key',\n",
329-
" content_type='application/x-recordio-protobuf',\n",
330-
")\n",
247+
"rcf = RandomCutForest(role=execution_role,\n",
248+
" train_instance_count=1,\n",
249+
" train_instance_type='ml.m4.xlarge',\n",
250+
" data_location='s3://{}/{}/'.format(bucket, prefix),\n",
251+
" output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
252+
" num_samples_per_tree=512,\n",
253+
" num_trees=50)\n",
331254
"\n",
332-
"# run the training job on input data stored in S3\n",
333-
"rcf.fit({'train': s3_train_input})"
255+
"# automatically upload the training data to S3 and run the training job\n",
256+
"rcf.fit(rcf.record_set(taxi_data.value.as_matrix().reshape(-1,1)))"
334257
]
335258
},
336259
{
@@ -408,15 +331,14 @@
408331
{
409332
"cell_type": "code",
410333
"execution_count": null,
411-
"metadata": {
412-
"collapsed": true
413-
},
334+
"metadata": {},
414335
"outputs": [],
415336
"source": [
416337
"from sagemaker.predictor import csv_serializer, json_deserializer\n",
417338
"\n",
418339
"rcf_inference.content_type = 'text/csv'\n",
419340
"rcf_inference.serializer = csv_serializer\n",
341+
"rcf_inference.accept = 'application/json'\n",
420342
"rcf_inference.deserializer = json_deserializer"
421343
]
422344
},
@@ -436,9 +358,8 @@
436358
"outputs": [],
437359
"source": [
438360
"taxi_data_numpy = taxi_data.value.as_matrix().reshape(-1,1)\n",
439-
"results = rcf_inference.predict(taxi_data_numpy[:6])\n",
440-
"\n",
441-
"print(results)"
361+
"print(taxi_data_numpy[:6])\n",
362+
"results = rcf_inference.predict(taxi_data_numpy[:6])"
442363
]
443364
},
444365
{
@@ -609,17 +530,11 @@
609530
" shingled_data[n] = data[n:(n+shingle_size)]\n",
610531
" return shingled_data\n",
611532
"\n",
612-
"# single data with shingle size=48 (one day) and upload to S3\n",
533+
"# single data with shingle size=48 (one day)\n",
613534
"shingle_size = 48\n",
614535
"prefix_shingled = 'sagemaker/randomcutforest_shingled'\n",
615536
"taxi_data_shingled = shingle(taxi_data.values[:,1], shingle_size)\n",
616-
"s3_train_data_shingled = convert_and_upload_training_data(\n",
617-
" taxi_data_shingled,\n",
618-
" bucket,\n",
619-
" prefix_shingled)\n",
620-
"\n",
621-
"print(taxi_data_shingled)\n",
622-
"print(s3_train_data_shingled)"
537+
"print(taxi_data_shingled)"
623538
]
624539
},
625540
{
@@ -637,28 +552,17 @@
637552
"source": [
638553
"session = sagemaker.Session()\n",
639554
"\n",
640-
"rcf = sagemaker.estimator.Estimator(\n",
641-
" container,\n",
642-
" execution_role,\n",
643-
" output_path='s3://{}/{}/output'.format(bucket, prefix_shingled),\n",
644-
" train_instance_count=1,\n",
645-
" train_instance_type='ml.m4.xlarge',\n",
646-
" sagemaker_session=session,\n",
647-
")\n",
648-
"\n",
649-
"rcf.set_hyperparameters(\n",
650-
" num_samples_per_tree = 200,\n",
651-
" num_trees = 50,\n",
652-
" feature_dim = shingle_size,\n",
653-
")\n",
654-
"\n",
655-
"s3_train_input = sagemaker.session.s3_input(\n",
656-
" s3_train_data_shingled,\n",
657-
" distribution='ShardedByS3Key',\n",
658-
" content_type='application/x-recordio-protobuf',\n",
659-
")\n",
555+
"# specify general training job information\n",
556+
"rcf = RandomCutForest(role=execution_role,\n",
557+
" train_instance_count=1,\n",
558+
" train_instance_type='ml.m4.xlarge',\n",
559+
" data_location='s3://{}/{}/'.format(bucket, prefix_shingled),\n",
560+
" output_path='s3://{}/{}/output'.format(bucket, prefix_shingled),\n",
561+
" num_samples_per_tree=512,\n",
562+
" num_trees=50)\n",
660563
"\n",
661-
"rcf.fit({'train': s3_train_input})"
564+
"# automatically upload the training data to S3 and run the training job\n",
565+
"rcf.fit(rcf.record_set(taxi_data_shingled))"
662566
]
663567
},
664568
{
@@ -676,6 +580,7 @@
676580
"\n",
677581
"rcf_inference.content_type = 'text/csv'\n",
678582
"rcf_inference.serializer = csv_serializer\n",
583+
"rcf_inference.accept = 'appliation/json'\n",
679584
"rcf_inference.deserializer = json_deserializer"
680585
]
681586
},
@@ -692,12 +597,13 @@
692597
"metadata": {},
693598
"outputs": [],
694599
"source": [
600+
"# Score the shingled datapoints\n",
695601
"results = rcf_inference.predict(taxi_data_shingled)\n",
696602
"scores = np.array([datum['score'] for datum in results['scores']])\n",
697603
"\n",
698604
"# compute the shingled score distribution and cutoff and determine anomalous scores\n",
699-
"score_mean = taxi_data.score.mean()\n",
700-
"score_std = taxi_data.score.std()\n",
605+
"score_mean = scores.mean()\n",
606+
"score_std = scores.std()\n",
701607
"score_cutoff = score_mean + 3*score_std\n",
702608
"\n",
703609
"anomalies = scores[scores > score_cutoff]\n",
@@ -747,7 +653,7 @@
747653
"cell_type": "markdown",
748654
"metadata": {},
749655
"source": [
750-
"We see that with this particular shingle size, hyperparameter selection, and anomaly cutoff threshold that the shingled approach more clearly captures the major anomalous events: the spike at around t=6000 and the dip at around t=10000. In general, the number of trees, sample size, and anomaly score cutoff are all parameters that a data scientist may need experiment with in order to achieve desired results. The use of a labeled test dataset allows the used to obtain common accuracy metrics for anomaly detection algorithms. For more information about Amazon SageMaker Random Cut Forest see the [AWS Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/randomcutforest.html)."
656+
"We see that with this particular shingle size, hyperparameter selection, and anomaly cutoff threshold that the shingled approach more clearly captures the major anomalous events: the spike at around t=6000 and the dips at around t=9000 and t=10000. In general, the number of trees, sample size, and anomaly score cutoff are all parameters that a data scientist may need experiment with in order to achieve desired results. The use of a labeled test dataset allows the used to obtain common accuracy metrics for anomaly detection algorithms. For more information about Amazon SageMaker Random Cut Forest see the [AWS Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/randomcutforest.html)."
751657
]
752658
},
753659
{

0 commit comments

Comments
 (0)