change: Make repack step output path align with model repack path #3257

qidewenwhen · 2022-07-20T18:18:27Z

Description of changes:

Issue 1:
The repack step is doing model repack in pipeline execution time and currently the output path of the repacked model is the default path generated by the repack step (a training job under the hood).
This output path is not aligned with the compile time model repack path (generated in Python SDK) and consequently it is not able to be customized via the code_location argument in model class.
This PR makes the repack step output path to be aligned with the compile time model repack path:
- If code_location is given, the runtime repack output path should be: s3://<bucket_name>/<code_location>/<model_name>/......
- If code_location is None, the runtime repack output path is: s3://<bucket_name>/<model_name>/......
Issue 2: ValueError: instance_type should not be a pipeline variable in SKLearnProcessor #3201
As customer pointed out, the warning message given by image_uris.retrieve is confusing. This PR also add more information in the warning message to clarify.

Testing done: unit tests and integ test

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-07-20T18:46:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a169e47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-07-20T18:49:16Z

Codecov Report

Merging #3257 (9b30cf4) into master (011539a) will decrease coverage by 0.74%.
The diff coverage is 94.73%.

@@            Coverage Diff             @@
##           master    #3257      +/-   ##
==========================================
- Coverage   89.61%   88.86%   -0.75%     
==========================================
  Files         645      203     -442     
  Lines       54585    17986   -36599     
==========================================
- Hits        48914    15984   -32930     
+ Misses       5671     2002    -3669

Impacted Files	Coverage Δ
src/sagemaker/workflow/utilities.py	`92.04% <75.00%> (ø)`
src/sagemaker/image_uris.py	`98.51% <100.00%> (ø)`
src/sagemaker/model.py	`92.77% <100.00%> (ø)`
src/sagemaker/rl/estimator.py	`92.95% <100.00%> (ø)`
src/sagemaker/tensorflow/model.py	`95.14% <100.00%> (ø)`
src/sagemaker/workflow/model_step.py	`93.47% <100.00%> (ø)`
src/sagemaker/workflow/pipeline_context.py	`97.36% <100.00%> (ø)`
.../sagemaker/cli/compatibility/v2/modifiers/serde.py
...python3.9/site-packages/sagemaker/sparkml/model.py
...maker/cli/compatibility/v2/modifiers/predictors.py
... and 845 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 011539a...9b30cf4. Read the comment docs.

sagemaker-bot · 2022-07-20T19:06:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a169e47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-20T19:15:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a169e47
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-20T19:26:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a169e47
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-20T20:02:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ae15050
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-20T20:20:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ae15050
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-20T20:23:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ae15050
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-20T20:43:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ae15050
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/image_uris.py

sagemaker-bot · 2022-07-20T22:37:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ae15050
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T03:16:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 75a446c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T03:48:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 75a446c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T04:03:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 75a446c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T04:16:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 75a446c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T04:20:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 75a446c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T17:36:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9b30cf4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T17:41:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 9b30cf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T17:51:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9b30cf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T18:16:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9b30cf4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T18:19:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 9b30cf4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2022-07-26T20:58:32Z

@qidewenwhen units are failing for this PR change. LGTM otherwise.

qidewenwhen · 2022-07-26T22:02:06Z

@qidewenwhen units are failing for this PR change. LGTM otherwise.

Thanks @mufaddal-rohawala! Have updated in the new commit.

sagemaker-bot · 2022-07-26T22:29:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e3af42f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T22:51:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e3af42f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T23:20:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e3af42f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T23:21:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e3af42f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T23:38:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e3af42f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-27T01:14:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e3af42f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…s#3257) Co-authored-by: Dewen Qi <[email protected]>

qidewenwhen force-pushed the repack branch from a169e47 to ae15050 Compare July 20, 2022 19:34

jerrypeng7773 reviewed Jul 20, 2022

View reviewed changes

src/sagemaker/image_uris.py Outdated Show resolved Hide resolved

jerrypeng7773 approved these changes Jul 20, 2022

View reviewed changes

qidewenwhen force-pushed the repack branch from ae15050 to 75a446c Compare July 23, 2022 03:12

qidewenwhen force-pushed the repack branch 2 times, most recently from 49e1e3a to 9b30cf4 Compare July 26, 2022 17:04

mufaddal-rohawala approved these changes Jul 26, 2022

View reviewed changes

change: Make repack step output path align with model repack path

e3af42f

qidewenwhen force-pushed the repack branch from 9b30cf4 to e3af42f Compare July 26, 2022 22:01

mufaddal-rohawala merged commit cdd8cb5 into aws:master Jul 27, 2022

knikure pushed a commit to knikure/sagemaker-python-sdk that referenced this pull request Aug 4, 2022

change: Make repack step output path align with model repack path (aw…

1c7847f

…s#3257) Co-authored-by: Dewen Qi <[email protected]>

tuliocasagrande mentioned this pull request Aug 6, 2022

feature: Allow custom output for RepackModelStep #2804

Closed

9 tasks

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

change: Make repack step output path align with model repack path (aw…

8f78094

…s#3257) Co-authored-by: Dewen Qi <[email protected]>

change: Make repack step output path align with model repack path #3257

change: Make repack step output path align with model repack path #3257

Uh oh!

Conversation

qidewenwhen commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

Uh oh!

sagemaker-bot commented Jul 20, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

Uh oh!

mufaddal-rohawala commented Jul 26, 2022

Uh oh!

qidewenwhen commented Jul 26, 2022

Uh oh!

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 26, 2022

qidewenwhen commented Jul 20, 2022 •

edited

Loading

codecov-commenter commented Jul 20, 2022 •

edited

Loading