change: Add PipelineVariable annotation in framework models #3188

qidewenwhen · 2022-06-22T00:54:01Z

Description of changes: This PR does the following:

Add PipelineVariable annotation in framework models. The PipelineVariable annotation will be leveraged by a test mechanism: change: Implement test mechanism for Pipeline variables #3163 to scan and check the incompatible issues between Pipeline steps and downstream classes caused by pipeline variables.
Fix issues figured out by the test mechanism

Testing done: unit tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-06-22T01:20:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1caee74
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-06-22T01:20:23Z

Codecov Report

Merging #3188 (45f40bc) into master (38b4916) will decrease coverage by 0.39%.
The diff coverage is 96.01%.

@@            Coverage Diff             @@
##           master    #3188      +/-   ##
==========================================
- Coverage   89.40%   89.01%   -0.40%     
==========================================
  Files         848      203     -645     
  Lines       72563    18232   -54331     
==========================================
- Hits        64877    16229   -48648     
+ Misses       7686     2003    -5683

Impacted Files	Coverage Δ
src/sagemaker/inputs.py	`100.00% <ø> (ø)`
src/sagemaker/jumpstart/types.py	`94.37% <0.00%> (ø)`
src/sagemaker/workflow/step_collections.py	`93.87% <ø> (ø)`
src/sagemaker/job.py	`90.00% <75.00%> (-0.55%)`	⬇️
src/sagemaker/chainer/model.py	`96.49% <88.88%> (-1.07%)`	⬇️
src/sagemaker/mxnet/model.py	`98.41% <90.00%> (+2.04%)`	⬆️
src/sagemaker/pytorch/model.py	`96.82% <90.00%> (+11.37%)`	⬆️
src/sagemaker/sklearn/model.py	`94.73% <90.00%> (+13.10%)`	⬆️
src/sagemaker/huggingface/model.py	`89.87% <92.85%> (+10.76%)`	⬆️
src/sagemaker/jumpstart/cache.py	`95.68% <93.54%> (-0.90%)`	⬇️
... and 689 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4afe7f...45f40bc. Read the comment docs.

sagemaker-bot · 2022-06-22T01:41:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1caee74
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T01:53:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 038c178
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T02:02:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 1caee74
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T02:13:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 038c178
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T02:34:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 038c178
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T02:45:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 038c178
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T02:54:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 038c178
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-06-22T03:47:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 1caee74
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T20:58:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a356d7d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:02:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: de6849d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:17:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: de6849d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:18:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a356d7d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:20:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a356d7d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:20:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: de6849d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:23:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: de6849d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:35:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a356d7d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-01T21:52:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: de6849d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T01:50:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 22746e1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:05:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 22746e1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:10:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 22746e1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:19:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 22746e1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:31:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 22746e1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T17:25:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 22746e1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T01:12:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 45f40bc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T01:40:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 45f40bc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T01:52:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 45f40bc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T02:12:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 45f40bc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T02:22:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 45f40bc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T04:13:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 45f40bc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T04:53:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 45f40bc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-31T01:03:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 45f40bc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Model annotate update Adjust as per recent model changes

sagemaker-bot · 2022-08-01T20:48:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 345033b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T21:09:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 345033b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T21:11:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 345033b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T21:22:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 345033b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni · 2022-08-01T21:28:34Z

src/sagemaker/pipeline.py

@@ -345,7 +346,7 @@ def register(
                framework_version=framework_version,
                nearest_model_name=nearest_model_name,
                data_input_configuration=data_input_configuration,
-                container_def=container_def,
+                container_list=container_def,


is it supposed to be a container_list. you are using container_def below

Yes, this should be a container_list. I used the Pipeline variable test mechanism (pending internal team review in #3163) to scan through, and found out that the container_def=container_def, would cause a bug.

The reason is PipelineModel here is actually a list of models and container_def generated by below is actually a list of container definitions

container_def = self.pipeline_container_def( inference_instances[0] if inference_instances else None )

Therefore, if we assign the list of container_def to container_def (like previously), it would break here with error AttributeError: 'list' object has no attribute 'update'

sagemaker-bot · 2022-08-01T21:31:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 345033b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Dewen Qi <[email protected]>

qidewenwhen marked this pull request as draft June 22, 2022 00:54

qidewenwhen force-pushed the fw_model_anno branch from 1caee74 to 038c178 Compare June 22, 2022 01:46

qidewenwhen force-pushed the fw_model_anno branch 2 times, most recently from a356d7d to de6849d Compare July 1, 2022 20:36

qidewenwhen force-pushed the fw_model_anno branch 2 times, most recently from fca5720 to 22746e1 Compare July 28, 2022 01:22

qidewenwhen marked this pull request as ready for review July 28, 2022 01:22

mufaddal-rohawala requested a review from navinsoni July 28, 2022 16:21

qidewenwhen force-pushed the fw_model_anno branch from 22746e1 to 45f40bc Compare July 29, 2022 01:02

aoguo64 approved these changes Aug 1, 2022

View reviewed changes

change: Add PipelineVariable annotation in framework models

345033b

Model annotate update Adjust as per recent model changes

navinsoni force-pushed the fw_model_anno branch from 45f40bc to 345033b Compare August 1, 2022 20:20

navinsoni reviewed Aug 1, 2022

View reviewed changes

navinsoni approved these changes Aug 1, 2022

View reviewed changes

navinsoni merged commit 60872f3 into aws:master Aug 1, 2022

knikure pushed a commit to knikure/sagemaker-python-sdk that referenced this pull request Aug 4, 2022

change: Add PipelineVariable annotation in framework models (aws#3188)

fc2c946

Co-authored-by: Dewen Qi <[email protected]>

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

change: Add PipelineVariable annotation in framework models (aws#3188)

2a40003

Co-authored-by: Dewen Qi <[email protected]>

jjerphan mentioned this pull request Nov 23, 2023

Specifying image_uri in PyTorchModel gives TypeError when running deploy #2202

Closed

qidewenwhen deleted the fw_model_anno branch January 25, 2024 18:34

change: Add PipelineVariable annotation in framework models #3188

change: Add PipelineVariable annotation in framework models #3188

Conversation

qidewenwhen commented Jun 22, 2022

Merge Checklist

General

Tests

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

codecov-commenter commented Jun 22, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jun 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

codecov-commenter commented Jun 22, 2022 •

edited

Loading