fix: Allow StepCollection added in ConditionStep to be depended on #3261

qidewenwhen · 2022-07-22T04:09:20Z

Issue #, if available:
Steps defined in ConditionStep are missed when building the step map in Pipeline object. Consequently, if there're StepCollections added in a ConditionStep and they depend on each other, a KeyError would be thrown due to the StepCollections are missed to add to the step map

Description of changes: Allow StepCollection added in ConditionStep to be depended on

Testing done: Unit tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-07-22T04:13:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6874e78
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T04:15:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6874e78
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T04:45:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6874e78
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T04:54:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6874e78
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T05:15:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6874e78
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T19:19:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6874e78
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T20:03:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a7ef598
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-07-22T20:05:03Z

Codecov Report

Merging #3261 (f46782f) into master (38b4916) will decrease coverage by 0.78%.
The diff coverage is 95.65%.

❗ Current head f46782f differs from pull request most recent head 5075346. Consider uploading reports for the commit 5075346 to get more accurate results

@@            Coverage Diff             @@
##           master    #3261      +/-   ##
==========================================
- Coverage   89.40%   88.62%   -0.79%     
==========================================
  Files         848      203     -645     
  Lines       72563    17982   -54581     
==========================================
- Hits        64877    15936   -48941     
+ Misses       7686     2046    -5640

Impacted Files	Coverage Δ
src/sagemaker/workflow/pipeline.py	`90.53% <95.65%> (-4.03%)`	⬇️
src/sagemaker/workflow/lambda_step.py	`89.83% <0.00%> (-5.09%)`	⬇️
src/sagemaker/workflow/execution_variables.py	`95.45% <0.00%> (-4.55%)`	⬇️
src/sagemaker/workflow/steps.py	`91.31% <0.00%> (-3.65%)`	⬇️
src/sagemaker/clarify.py	`93.80% <0.00%> (-1.24%)`	⬇️
src/sagemaker/workflow/properties.py	`88.09% <0.00%> (-1.20%)`	⬇️
...hon3.7/site-packages/sagemaker/tensorflow/model.py
...python3.7/site-packages/sagemaker/lineage/query.py
...ython3.7/site-packages/sagemaker/local/entities.py
... and 642 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 79b3e9d...5075346. Read the comment docs.

sagemaker-bot · 2022-07-22T20:21:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a7ef598
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T20:23:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a7ef598
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T20:29:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a7ef598
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T20:40:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a7ef598
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T22:40:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f46782f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T22:48:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f46782f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T23:05:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f46782f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T23:17:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f46782f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-22T23:21:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f46782f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T00:38:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f46782f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T01:50:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ccc494e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T02:31:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a377b72
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T02:37:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a377b72
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T02:50:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a377b72
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T03:09:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a377b72
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-23T03:17:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a377b72
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

qidewenwhen · 2022-07-26T22:22:55Z

src/sagemaker/workflow/pipeline.py


-@attr.s


Removing the @attr.s and define __init__ directly, as the former is not thread safe.
With the use of @attr.s, multiple unit tests of the PR failed due to ValueError("Pipeline steps cannot have duplicate names."), when invoking _generate_step_map(self.steps, self._step_map)
The reason is, in this case, in multiple unit tests, _generate_step_map methods are building the same global _step_map. As a result, if a step name duplicates in different tests, it can cause the ValueError("Pipeline steps cannot have duplicate names.").

sagemaker-bot · 2022-07-27T19:37:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 30051c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-27T19:44:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 30051c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-27T19:55:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 30051c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-27T20:16:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 30051c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-27T20:20:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 30051c7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:36:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5075346
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:42:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 5075346
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T02:55:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5075346
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T03:17:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5075346
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T03:19:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5075346
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ws#3261) Co-authored-by: Dewen Qi <[email protected]>

qidewenwhen marked this pull request as draft July 22, 2022 04:09

qidewenwhen force-pushed the dev branch from 6874e78 to a7ef598 Compare July 22, 2022 19:34

qidewenwhen marked this pull request as ready for review July 22, 2022 19:40

qidewenwhen force-pushed the dev branch from a7ef598 to f46782f Compare July 22, 2022 22:13

qidewenwhen force-pushed the dev branch from f46782f to ccc494e Compare July 23, 2022 01:44

qidewenwhen force-pushed the dev branch from ccc494e to a377b72 Compare July 23, 2022 02:01

qidewenwhen commented Jul 26, 2022

View reviewed changes

staubhp approved these changes Jul 26, 2022

View reviewed changes

nmadan approved these changes Jul 27, 2022

View reviewed changes

qidewenwhen force-pushed the dev branch from a377b72 to 30051c7 Compare July 27, 2022 19:07

nmadan approved these changes Jul 27, 2022

View reviewed changes

staubhp approved these changes Jul 27, 2022

View reviewed changes

fix: Allow StepCollection added in ConditionStep to be depended on

5075346

qidewenwhen force-pushed the dev branch from 30051c7 to 5075346 Compare July 28, 2022 02:06

navinsoni approved these changes Aug 1, 2022

View reviewed changes

navinsoni merged commit 2d59111 into aws:master Aug 1, 2022

knikure pushed a commit to knikure/sagemaker-python-sdk that referenced this pull request Aug 4, 2022

fix: Allow StepCollection added in ConditionStep to be depended on (a…

dca9c4a

…ws#3261) Co-authored-by: Dewen Qi <[email protected]>

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

fix: Allow StepCollection added in ConditionStep to be depended on (a…

5d8b65c

…ws#3261) Co-authored-by: Dewen Qi <[email protected]>


		@attr.s

fix: Allow StepCollection added in ConditionStep to be depended on #3261

fix: Allow StepCollection added in ConditionStep to be depended on #3261

Conversation

qidewenwhen commented Jul 22, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

codecov-commenter commented Jul 22, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2022

AWS CodeBuild CI Report

qidewenwhen Jul 26, 2022

Choose a reason for hiding this comment

sagemaker-bot commented Jul 27, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 27, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 27, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 27, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 27, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

qidewenwhen commented Jul 22, 2022 •

edited

Loading

codecov-commenter commented Jul 22, 2022 •

edited

Loading