feature: support large pipeline #2825

navaj0 · 2022-01-05T23:10:08Z

Issue #, if available:

Description of changes:

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter · 2022-01-05T23:24:13Z

Codecov Report

Merging #2825 (f007d03) into dev (8e9d9b7) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              dev    #2825      +/-   ##
==========================================
+ Coverage   88.69%   88.70%   +0.01%     
==========================================
  Files         174      175       +1     
  Lines       15461    15477      +16     
==========================================
+ Hits        13713    13729      +16     
  Misses       1748     1748

Impacted Files	Coverage Δ
src/sagemaker/workflow/parallelism_config.py	`100.00% <100.00%> (ø)`
src/sagemaker/workflow/pipeline.py	`89.26% <100.00%> (+0.60%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e9d9b7...f007d03. Read the comment docs.

sagemaker-bot · 2022-01-05T23:28:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 0b30765
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T00:09:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 0b30765
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T00:20:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0b30765
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T00:23:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 0b30765
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T01:03:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ef9f006
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/workflow/pipeline.py

sagemaker-bot · 2022-01-06T01:28:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 0b30765
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T01:36:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ef9f006
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T01:49:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ef9f006
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T02:12:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ef9f006
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T02:29:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ef9f006
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…intion

…hods

sagemaker-bot · 2022-01-06T20:12:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 28afb1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T20:50:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d360659
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:01:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 28afb1e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:15:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 28afb1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:16:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d360659
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jeniyat · 2022-01-06T21:24:44Z

src/sagemaker/workflow/pipeline.py

        update_args(
            kwargs,
            Tags=tags,
        )
        return self.sagemaker_session.sagemaker_client.create_pipeline(**kwargs)

-    def _create_args(self, role_arn: str, description: str):
+    def _create_args(
+        self, role_arn: str, description: str, parallelism_config: ParallelismConfiguration


Is it possible to add default values to these parameters?

Actually no. All of them must be explicitly set by the users.

jeniyat · 2022-01-06T21:26:54Z

tests/integ/test_workflow.py

+    callback_steps = [
+        CallbackStep(
+            name=f"callback-step{count}",
+            sqs_queue_url="https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue",


Will it be an accessible url even if the user is from a different region (i.e., us-west-2)?

It is not an accessible url. It is actually just a well formatted dummy URL embedded in the pipeline definition to verify the pipeline creation.

sagemaker-bot · 2022-01-06T21:30:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d360659
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:30:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d360659
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-06T21:42:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 28afb1e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T19:59:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f69a544
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T20:34:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f69a544
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T20:42:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f69a544
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T21:44:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f007d03
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T21:57:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f007d03
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T22:25:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f007d03
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-07T22:36:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f007d03
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Ameen Khan <[email protected]> Co-authored-by: Zhankui Lu <[email protected]> Co-authored-by: Ahsan Khan <[email protected]>

staubhp approved these changes Jan 5, 2022

View reviewed changes

navaj0 changed the title ~~Feature/large pipeline~~ feature: support large pipeline Jan 6, 2022

mufaddal-rohawala requested changes Jan 6, 2022

View reviewed changes

src/sagemaker/workflow/pipeline.py Outdated Show resolved Hide resolved

src/sagemaker/workflow/pipeline.py Show resolved Hide resolved

Ameen Khan added 4 commits January 6, 2022 11:53

feature: Updated create/update pipeline to support large pipeline def…

e9f59a3

…intion

feature: Added parallelism config to create/update/start pipeline met…

04396ff

…hods

Updated docstring, added describe call to check config

5afcfab

Fixed formatting issues

eac9d22

navaj0 force-pushed the feature/large_pipeline branch from ef9f006 to 28afb1e Compare January 6, 2022 20:05

Add missing unit tests for s3 upload

d360659

navaj0 force-pushed the feature/large_pipeline branch from 28afb1e to d360659 Compare January 6, 2022 20:33

jeniyat suggested changes Jan 6, 2022

View reviewed changes

jeniyat approved these changes Jan 6, 2022

View reviewed changes

Merge branch 'dev' into feature/large_pipeline

f69a544

mufaddal-rohawala approved these changes Jan 7, 2022

View reviewed changes

Merge branch 'dev' into feature/large_pipeline

f007d03

ahsan-z-khan merged commit b18435c into aws:dev Jan 7, 2022

ameenkhan07 mentioned this pull request Feb 7, 2022

Feature/large pipeline #2706

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: support large pipeline #2825

feature: support large pipeline #2825

navaj0 commented Jan 5, 2022

codecov-commenter commented Jan 5, 2022 •

edited

Loading

sagemaker-bot commented Jan 5, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

jeniyat Jan 6, 2022

navaj0 Jan 6, 2022

jeniyat Jan 6, 2022

navaj0 Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 6, 2022

sagemaker-bot commented Jan 7, 2022

sagemaker-bot commented Jan 7, 2022

sagemaker-bot commented Jan 7, 2022

sagemaker-bot commented Jan 7, 2022

sagemaker-bot commented Jan 7, 2022

sagemaker-bot commented Jan 7, 2022

sagemaker-bot commented Jan 7, 2022

feature: support large pipeline #2825

feature: support large pipeline #2825

Conversation

navaj0 commented Jan 5, 2022

Merge Checklist

General

Tests

codecov-commenter commented Jan 5, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Jan 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

jeniyat Jan 6, 2022

Choose a reason for hiding this comment

navaj0 Jan 6, 2022

Choose a reason for hiding this comment

jeniyat Jan 6, 2022

Choose a reason for hiding this comment

navaj0 Jan 6, 2022

Choose a reason for hiding this comment

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 6, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 7, 2022

AWS CodeBuild CI Report

codecov-commenter commented Jan 5, 2022 •

edited

Loading