feature: support inter-container traffic encryption for processing jobs #1431

szeinedd · 2020-04-21T00:04:09Z

Issue #, if available:
#1320

Description of changes:
Adding support for inter container traffic encryption for processing jobs

[DO NOT MERGE] until May 8th.

Testing done:

unit tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
[N/A] I have passed the region in to any/all clients that I've initialized as part of this change.
[N/A] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
[N/A] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2020-04-21T00:13:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-21T00:25:01Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-21T00:38:25Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-21T00:49:46Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-22T05:45:31Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-22T05:57:48Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-22T06:08:40Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-22T06:09:44Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/model_monitor/model_monitoring.py

tests/unit/test_processing.py

laurenyu · 2020-04-22T23:34:04Z

also, can you add an integ test?

szeinedd · 2020-04-27T04:55:21Z

also, can you add an integ test?

Added integ test, passed locally.

tests/integ/test_processing.py::test_sklearn_with_network_config
Job Name: test-sklearn-with-network-config-2020-04-27-04-31-22-387
Inputs: [{'InputName': 'input-1', 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-304265168256/test-sklearn-with-network-config-2020-04-27-04-31-22-387/input/input-1/dummy_input.txt', 'LocalPath': '/opt/ml/processing/inputs/', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'code', 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-304265168256/test-sklearn-with-network-config-2020-04-27-04-31-22-387/input/code/dummy_script.py', 'LocalPath': '/opt/ml/processing/input/code', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]
Outputs: []
PASSED

sagemaker-bot · 2020-04-27T04:56:16Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-27T05:11:22Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-27T05:20:15Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-04-27T05:20:23Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

laurenyu

just some small comments. otherwise looks good!

laurenyu · 2020-04-28T21:31:21Z

src/sagemaker/model_monitor/model_monitoring.py

@@ -961,6 +963,23 @@ def _wait_for_schedule_changes_to_apply(self):
            if schedule_desc["MonitoringScheduleStatus"] != "Pending":
                break

+    def validate_network_config(self, network_config_dict):


let's make this a private method so that it can be easily removed in the future

laurenyu · 2020-04-28T21:31:43Z

src/sagemaker/model_monitor/model_monitoring.py

+    def validate_network_config(self, network_config_dict):
+        """Validates that EnableInterContainerTrafficEncryption is not set in
+        NetworkConfig object
+


docstring is missing the "Args" section

laurenyu · 2020-04-28T21:32:35Z

src/sagemaker/model_monitor/model_monitoring.py

+                    self.network_config.encrypt_inter_container_traffic
+                )
+            )
+            print(message)


use a logger instead of a print statement so that the user can control what log messages are printed to the console

laurenyu · 2020-04-28T21:33:57Z

src/sagemaker/network.py

+            encrypt_inter_container_traffic (bool): Boolean that determines whether to
+                encrypt inter-container traffic.


can you also add a sentence about what the default behavior is?

also, move this down to the end of the docstring so that the order matches the order of the parameters

sagemaker-bot · 2020-05-02T01:50:46Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-02T02:07:28Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-02T02:15:16Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-02T02:19:44Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:02:18Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:12:55Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:24:45Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-05T21:59:30Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-07T16:12:41Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-07T16:29:10Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-07T16:34:37Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-07T16:35:06Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chuyang-deng · 2020-05-08T21:23:01Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chuyang-deng · 2020-05-08T21:23:01Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chuyang-deng · 2020-05-08T21:23:02Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:23:20Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:34:16Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:44:23Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:44:52Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:55:11Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:56:17Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-05-08T21:56:31Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chuyang-deng changed the title ~~feature: support inter container traffic encryption for processing jobs~~ [DO NOT MERGE] feature: support inter container traffic encryption for processing jobs Apr 21, 2020

szeinedd force-pushed the processing-jobs-cie branch from 9478a6f to c97d088 Compare April 22, 2020 05:36

laurenyu reviewed Apr 22, 2020

View reviewed changes

src/sagemaker/model_monitor/model_monitoring.py Outdated Show resolved Hide resolved

tests/unit/test_processing.py Show resolved Hide resolved

szeinedd force-pushed the processing-jobs-cie branch from c97d088 to aed0dc7 Compare April 27, 2020 04:46

laurenyu reviewed Apr 28, 2020

View reviewed changes

feature: support inter container traffic encryption for processing jobs

c61d899

szeinedd force-pushed the processing-jobs-cie branch from aed0dc7 to c61d899 Compare May 2, 2020 01:46

laurenyu approved these changes May 5, 2020

View reviewed changes

Merge branch 'master' into processing-jobs-cie

8705ecd

Merge branch 'master' into processing-jobs-cie

9583095

laurenyu mentioned this pull request May 7, 2020

The processing feature does not allow use of encrypt_inter_container_traffic parameter #1320

Closed

Merge branch 'master' into processing-jobs-cie

34ba78a

laurenyu changed the title ~~[DO NOT MERGE] feature: support inter container traffic encryption for processing jobs~~ feature: support inter-container traffic encryption for processing jobs May 8, 2020

laurenyu merged commit 03bc337 into aws:master May 8, 2020

laurenyu mentioned this pull request May 9, 2020

fix: upgrade boto3 to 1.13.6 #1483

Merged

7 tasks

		encrypt_inter_container_traffic (bool): Boolean that determines whether to
		encrypt inter-container traffic.

feature: support inter-container traffic encryption for processing jobs #1431

feature: support inter-container traffic encryption for processing jobs #1431

Uh oh!

Conversation

szeinedd commented Apr 21, 2020 • edited by laurenyu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Apr 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 22, 2020

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

laurenyu commented Apr 22, 2020

Uh oh!

szeinedd commented Apr 27, 2020

Uh oh!

sagemaker-bot commented Apr 27, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Apr 27, 2020

AWS CodeBuild CI Report

Uh oh!

laurenyu left a comment

Choose a reason for hiding this comment

Uh oh!

laurenyu Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

szeinedd May 2, 2020

Choose a reason for hiding this comment

Uh oh!

laurenyu Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

szeinedd May 2, 2020

Choose a reason for hiding this comment

Uh oh!

laurenyu Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

laurenyu Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

szeinedd May 2, 2020

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented May 2, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented May 2, 2020

szeinedd commented Apr 21, 2020 •

edited by laurenyu

Loading