Implement subclass compatibility for workflow pipeline job steps #3040

jerrypeng7773 · 2022-03-31T17:58:40Z

Issue #, if available:

Description of changes:

Implement subclass compatibility for workflow pipeline job steps. Please reach out to the Pipeline team before merge, there are few other PRs we would like to release together.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-03-31T18:10:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: deeb7c1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T18:24:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5616ba7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T18:29:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: deeb7c1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T18:43:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5616ba7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T19:50:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5616ba7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T20:16:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 86b0a8f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T20:43:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 27169d2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-03-31T20:50:44Z

Codecov Report

Merging #3040 (9333921) into master (6ad4e45) will increase coverage by 0.08%.
The diff coverage is 97.45%.

@@            Coverage Diff             @@
##           master    #3040      +/-   ##
==========================================
+ Coverage   89.44%   89.53%   +0.08%     
==========================================
  Files         198      199       +1     
  Lines       17077    17174      +97     
==========================================
+ Hits        15275    15376     +101     
+ Misses       1802     1798       -4

Impacted Files	Coverage Δ
src/sagemaker/workflow/steps.py	`96.77% <93.75%> (-1.19%)`	⬇️
src/sagemaker/amazon/amazon_estimator.py	`89.51% <100.00%> (+0.14%)`	⬆️
src/sagemaker/clarify.py	`95.47% <100.00%> (+2.46%)`	⬆️
src/sagemaker/estimator.py	`91.46% <100.00%> (+0.16%)`	⬆️
src/sagemaker/processing.py	`95.83% <100.00%> (-0.28%)`	⬇️
src/sagemaker/session.py	`70.93% <100.00%> (+0.41%)`	⬆️
src/sagemaker/spark/processing.py	`98.91% <100.00%> (ø)`
src/sagemaker/transformer.py	`99.34% <100.00%> (+0.01%)`	⬆️
src/sagemaker/tuner.py	`98.13% <100.00%> (+<0.01%)`	⬆️
src/sagemaker/workflow/pipeline_context.py	`100.00% <100.00%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ad4e45...9333921. Read the comment docs.

sagemaker-bot · 2022-03-31T21:18:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 86b0a8f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T21:18:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 27169d2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T21:49:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 86b0a8f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T21:49:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 27169d2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-08T20:19:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9afec64
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-08T21:25:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9afec64
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/workflow/steps.py

sagemaker-bot · 2022-04-13T23:59:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7c4bd16
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-13T23:59:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8ad11ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-14T00:33:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7c4bd16
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/processing.py

src/sagemaker/workflow/steps.py

src/sagemaker/transformer.py

src/sagemaker/workflow/pipeline_context.py

sagemaker-bot · 2022-04-19T16:45:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6cd6c42
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-22T20:28:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 67f60b7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-22T20:43:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 67f60b7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-22T20:58:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 67f60b7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-22T22:46:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 67f60b7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…2935)

Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: HappyAmazonian <[email protected]>

Co-authored-by: Dewen Qi <[email protected]>

sagemaker-bot · 2022-04-25T16:59:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 46275e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T17:17:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 46275e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T17:23:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 46275e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T17:29:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 46275e8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T17:44:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 46275e8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T19:10:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 46275e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T19:29:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 46275e8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-25T22:28:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 46275e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-26T19:32:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9333921
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-26T19:47:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 9333921
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-26T19:57:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9333921
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-26T20:14:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9333921
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-26T20:21:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 9333921
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…#3040) Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: HappyAmazonian <[email protected]> Co-authored-by: Dewen Qi <[email protected]> Co-authored-by: Madhubalasri B <[email protected]> Co-authored-by: evakravi <[email protected]>

jerrypeng7773 force-pushed the job-step-subclass branch from deeb7c1 to 5616ba7 Compare March 31, 2022 18:13

jerrypeng7773 changed the title ~~Job step subclass~~ Implement subclass compatibility for workflow pipeline job steps Mar 31, 2022

jerrypeng7773 force-pushed the job-step-subclass branch from 5616ba7 to 86b0a8f Compare March 31, 2022 20:02

jerrypeng7773 force-pushed the job-step-subclass branch from 86b0a8f to 27169d2 Compare March 31, 2022 20:20

xchen909 added the component: pipelines Relates to the SageMaker Pipeline Platform label Apr 4, 2022

jerrypeng7773 force-pushed the job-step-subclass branch from 27169d2 to 9afec64 Compare April 8, 2022 19:47

qidewenwhen reviewed Apr 13, 2022

View reviewed changes

src/sagemaker/workflow/steps.py Show resolved Hide resolved

qidewenwhen requested review from qidewenwhen and removed request for qidewenwhen April 13, 2022 22:19

jerrypeng7773 force-pushed the job-step-subclass branch from 9afec64 to 8ad11ab Compare April 13, 2022 23:30

qidewenwhen reviewed Apr 19, 2022

View reviewed changes

jerrypeng7773 force-pushed the job-step-subclass branch from 6cd6c42 to abbf0ca Compare April 19, 2022 17:48

MadhubalasriB and others added 4 commits April 25, 2022 09:39

feature: adding customer metadata support to registermodel step (aws#…

bd03840

…2935)

feat: override jumpstart content bucket (aws#2901)

19c4df6

Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: HappyAmazonian <[email protected]>

fix: Improve Pipeline workflow unit test branch coverage (aws#2878)

125cdcf

Co-authored-by: Dewen Qi <[email protected]>

Implement subclass compatibility for workflow pipeline job steps

46275e8

jerrypeng7773 force-pushed the job-step-subclass branch from 67f60b7 to 46275e8 Compare April 25, 2022 16:39

qidewenwhen approved these changes Apr 25, 2022

View reviewed changes

navaj0 approved these changes Apr 26, 2022

View reviewed changes

ahsan-z-khan approved these changes Apr 26, 2022

View reviewed changes

Merge branch 'master' into job-step-subclass

9333921

mufaddal-rohawala approved these changes Apr 27, 2022

View reviewed changes

shreyapandit merged commit c46a09d into aws:master Apr 27, 2022

Implement subclass compatibility for workflow pipeline job steps #3040

Implement subclass compatibility for workflow pipeline job steps #3040

Conversation

jerrypeng7773 commented Mar 31, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

codecov-commenter commented Mar 31, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 14, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 26, 2022

AWS CodeBuild CI Report

jerrypeng7773 commented Mar 31, 2022 •

edited

Loading

codecov-commenter commented Mar 31, 2022 •

edited

Loading