feature: Combined inference and training script artifact #3717

evakravi · 2023-03-14T15:09:50Z

Issue #, if available:

Description of changes:
`script_uris.retrieve returns a combined inference + training script artifact if available in the metadata.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-03-14T16:30:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c09435f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-14T16:36:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c09435f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-14T16:38:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c09435f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-14T16:42:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: c09435f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-14T18:42:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c09435f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-03-15T21:24:54Z

Codecov Report

Merging #3717 (35db230) into master (d81a305) will decrease coverage by 0.76%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3717      +/-   ##
==========================================
- Coverage   90.19%   89.44%   -0.76%     
==========================================
  Files        1000      234     -766     
  Lines       94824    23460   -71364     
==========================================
- Hits        85524    20983   -64541     
+ Misses       9300     2477    -6823

Impacted Files	Coverage Δ
src/sagemaker/jumpstart/artifacts.py	`90.19% <100.00%> (ø)`
src/sagemaker/jumpstart/types.py	`95.20% <100.00%> (ø)`
src/sagemaker/utils.py	`94.83% <100.00%> (ø)`

... and 1231 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-03-15T22:26:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 80b0464
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-15T22:31:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 80b0464
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-15T22:32:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 80b0464
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-15T22:35:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 80b0464
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-16T00:40:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 80b0464
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

JGuinegagne · 2023-03-16T14:30:16Z

src/sagemaker/jumpstart/artifacts.py

-        model_script_key = model_specs.hosting_script_key
+        model_script_key = (
+            getattr(model_specs, "training_prepacked_script_key") or model_specs.hosting_script_key
+        )


question: shouldn't there be a way to access the uncombined model artifact?

Could you please align with @whittech1 on this?

We decided that there is no harm in including the script with the model artifact, even if someone only wants the model artifact. Most of the combined tarball is the model artifact, so it doesn't make a huge difference in terms of storage

The risk is that there are extra files left over from the combined model artifact in the code dir when a user tries to override the script. In the worst case, it could make the new combined script crash. Maybe we should sleep on this.

sagemaker-bot · 2023-03-20T15:29:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 72cc8ea
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-20T15:32:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 72cc8ea
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-20T15:37:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 72cc8ea
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-20T15:39:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 72cc8ea
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-20T17:42:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 72cc8ea
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-21T14:56:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 26ea2eb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T15:07:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4f345db
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T15:36:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4f345db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T15:56:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4f345db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T15:57:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4f345db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T16:01:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4f345db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T17:46:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0841f13
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T18:30:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0841f13
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T18:31:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 0841f13
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T18:36:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0841f13
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-03T18:40:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0841f13
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T16:00:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T16:04:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T16:06:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T16:12:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T16:24:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T21:55:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T21:55:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T21:55:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T21:55:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-04-04T22:55:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 35db230
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

feat: combined inference + training script artifact

edcfe67

evakravi requested a review from a team as a code owner March 14, 2023 15:09

evakravi requested review from trajanikant and removed request for a team March 14, 2023 15:09

evakravi marked this pull request as draft March 14, 2023 15:09

fix: jumpstart unit tests

c09435f

chore: always include training script if available

80b0464

JGuinegagne reviewed Mar 16, 2023

View reviewed changes

JGuinegagne approved these changes Mar 17, 2023

View reviewed changes

Merge branch 'master' into feat/inference-plus-training-script

72cc8ea

whittech1 approved these changes Mar 20, 2023

View reviewed changes

evakravi marked this pull request as ready for review March 21, 2023 14:51

Merge branch 'master' into feat/inference-plus-training-script

26ea2eb

Merge branch 'master' into feat/inference-plus-training-script

0841f13

evakravi marked this pull request as draft April 4, 2023 15:00

evakravi added 2 commits April 4, 2023 15:09

fix: script scope should be training

8e461ca

chore: remove extra line

35db230

evakravi marked this pull request as ready for review April 4, 2023 15:42

whittech1 approved these changes Apr 4, 2023

View reviewed changes

knikure approved these changes Apr 4, 2023

View reviewed changes

knikure changed the title ~~feat: combined inference + training script artifact~~ feature: Combined inference and training script artifact Apr 4, 2023

knikure merged commit 1045edd into aws:master Apr 4, 2023

doddaspk-amzn pushed a commit to doddaspk-amzn/sagemaker-python-sdk that referenced this pull request Apr 6, 2023

feature: Combined inference and training script artifact (aws#3717)

53fea69

feature: Combined inference and training script artifact #3717

feature: Combined inference and training script artifact #3717

Conversation

evakravi commented Mar 14, 2023 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Mar 14, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 14, 2023

AWS CodeBuild CI Report

codecov-commenter commented Mar 15, 2023 • edited Loading

Codecov Report

sagemaker-bot commented Mar 15, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 15, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 15, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 15, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 16, 2023

AWS CodeBuild CI Report

JGuinegagne Mar 16, 2023

Choose a reason for hiding this comment

evakravi Mar 17, 2023

Choose a reason for hiding this comment

JGuinegagne Mar 17, 2023

Choose a reason for hiding this comment

sagemaker-bot commented Mar 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 20, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 3, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2023

evakravi commented Mar 14, 2023 •

edited

Loading

codecov-commenter commented Mar 15, 2023 •

edited

Loading