feature: Allow setting S3 endpoint URL for Local Session #1359

larsll · 2020-03-14T14:43:15Z

Issue #, if available: #1358

Description of changes: Makes it possible to pass in a different S3 endpoint URL when instantiating a Local Session. This enables local jobs to connect to a locally hosted endpoint, e.g. Minio, in offline settings, or when it is not desirable to transfer large amounts of data into the cloud.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have used the regional endpoint when creating S3 and/or STS clients (if appropriate)
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2020-03-14T14:51:49Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:00:08Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:08:24Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:16:20Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:21:07Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:28:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:31:53Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:47:43Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:51:21Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-14T15:55:35Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chuyang-deng · 2020-03-16T17:10:34Z

Thanks for the contribution! Could you add unit tests for your changes and preferably one inte test as well?

The unit test can be added in to an existing framework training integ test file and using the local S3 url

src/sagemaker/estimator.py

sagemaker-bot · 2020-03-20T22:26:53Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-20T22:47:30Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-20T22:49:53Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-20T22:55:45Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-21T21:48:58Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-21T22:01:53Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-21T22:08:23Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-21T22:17:05Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-21T22:17:13Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-21T22:17:56Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-24T22:31:34Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-24T22:31:55Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/fw_utils.py

src/sagemaker/local/image.py

src/sagemaker/local/local_session.py

src/sagemaker/session.py

tests/unit/test_local_session.py

sagemaker-bot · 2020-03-24T22:36:25Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-Authored-By: Lauren Yu <[email protected]>

sagemaker-bot · 2020-03-25T22:33:33Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-25T22:53:41Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-25T23:01:03Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-25T23:03:07Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T19:34:33Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T19:39:25Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T19:49:25Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T19:55:00Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T20:26:38Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T20:26:56Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T20:31:36Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/local/local_session.py

sagemaker-bot · 2020-03-26T20:41:49Z

AWS CodeBuild CI Report

Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T20:51:59Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T21:05:56Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T21:16:09Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-03-26T21:41:28Z

AWS CodeBuild CI Report

Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

feature: Allow for pre-configured s3Client

803e052

fix: Documentation updated

5433ef5

fix: linting

6899f75

laurenyu reviewed Mar 16, 2020

View reviewed changes

src/sagemaker/estimator.py Outdated Show resolved Hide resolved

Renaming

29eb4ca

Refactor + fix tests

a4627f8

Formatting

4d70d9a

Removing unused code

c25cc4f

Adding unit test

78f188b

laurenyu reviewed Mar 24, 2020

View reviewed changes

Update src/sagemaker/fw_utils.py after review comment

022af2f

Co-Authored-By: Lauren Yu <[email protected]>

larsll changed the title ~~feature: Allow for pre-configured s3Client~~ feature: Allow setting S3 endpoint URL for Local Session Mar 26, 2020

larsll and others added 2 commits March 26, 2020 19:25

fix: code review updates

06a296f

Merge branch 'master' into s3client-config

52fbe55

larsll requested a review from laurenyu March 26, 2020 19:46

laurenyu reviewed Mar 26, 2020

View reviewed changes

src/sagemaker/local/local_session.py Show resolved Hide resolved

fix: test error in py27

27628c6

laurenyu approved these changes Mar 26, 2020

View reviewed changes

laurenyu merged commit d8b3012 into aws:master Mar 26, 2020

laurenyu mentioned this pull request May 7, 2020

Support using local S3 instance (e.g. minio) #1358

Closed

feature: Allow setting S3 endpoint URL for Local Session #1359

feature: Allow setting S3 endpoint URL for Local Session #1359

Conversation

larsll commented Mar 14, 2020 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2020

AWS CodeBuild CI Report

chuyang-deng commented Mar 16, 2020

sagemaker-bot commented Mar 20, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 25, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 25, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 25, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 25, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

sagemaker-bot commented Mar 26, 2020

AWS CodeBuild CI Report

larsll commented Mar 14, 2020 •

edited

Loading