fix: support idempotency for framework and spark processors #3460

brockwade633 · 2022-11-07T13:50:32Z

Issue #, if available:
#3451

Description of changes:
These changes address a bug described in more detail in the above GH issue. Provides idempotency support for FrameworkProcessors, PySparkProcessors, and SparkJarProcessors.

Testing done:

Extended processing, training, transform, and tuning step unit test suites to include assertions for idempotency
Verified no duplicate inputs in the pipeline definition from Jupyter notebook that exposed this issue

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-11-07T13:56:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 85d5be2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T14:35:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 85d5be2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T14:46:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 85d5be2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T14:46:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 85d5be2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T14:53:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 85d5be2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T16:23:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 645c938
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T16:29:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 645c938
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T16:40:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 645c938
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T16:43:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 645c938
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-07T16:51:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 645c938
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-11-10T21:19:25Z

Codecov Report

Merging #3460 (a7a8c38) into master (1fa2377) will decrease coverage by 0.18%.
The diff coverage is 88.76%.

@@            Coverage Diff             @@
##           master    #3460      +/-   ##
==========================================
- Coverage   89.17%   88.98%   -0.19%     
==========================================
  Files         204      212       +8     
  Lines       18979    20447    +1468     
==========================================
+ Hits        16924    18195    +1271     
- Misses       2055     2252     +197

Impacted Files	Coverage Δ
src/sagemaker/feature_store/feature_group.py	`76.52% <ø> (ø)`
src/sagemaker/huggingface/model.py	`89.87% <ø> (ø)`
src/sagemaker/pipeline.py	`85.33% <ø> (ø)`
src/sagemaker/sklearn/model.py	`92.98% <ø> (-1.76%)`	⬇️
...sagemaker/workflow/monitor_batch_transform_step.py	`0.00% <0.00%> (ø)`
src/sagemaker/xgboost/model.py	`96.36% <ø> (-1.82%)`	⬇️
...agemaker/model_monitor/clarify_model_monitoring.py	`89.84% <27.58%> (-5.09%)`	⬇️
src/sagemaker/training_compiler/config.py	`80.43% <60.00%> (-2.90%)`	⬇️
src/sagemaker/pytorch/estimator.py	`94.73% <63.63%> (-5.27%)`	⬇️
src/sagemaker/sklearn/estimator.py	`93.93% <63.63%> (-6.07%)`	⬇️
... and 50 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-11-10T21:28:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 864a90a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-10T22:07:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 864a90a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-10T22:15:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 864a90a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-10T22:20:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 864a90a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-10T22:24:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 864a90a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/spark/processing.py

mufaddal-rohawala

Units are failing for the changes in the PR @brockwade633 can you fix?

sagemaker-bot · 2022-12-01T04:46:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 7ffb430
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T04:46:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7ffb430
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T04:51:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7ffb430
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T05:15:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 7ffb430
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T05:20:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 7ffb430
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2022-12-01T17:42:39Z

@brockwade633 can you check the failing units.

sagemaker-bot · 2022-12-05T20:02:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T20:09:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T20:20:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T20:27:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a7a8c38
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T20:39:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knikure

/bot run notebook-tests

sagemaker-bot · 2022-12-05T22:04:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T22:48:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T22:51:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T23:18:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T23:19:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a7a8c38
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-05T23:23:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a7a8c38
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

bug-fix: support idempotency for framework and spark processors

85d5be2

brockwade633 mentioned this pull request Nov 7, 2022

multiple calls to pipeline.definition() causes extra "code" ProcessingInput to the appended to ProcessingStep #3451

Closed

change: update test path

645c938

nmadan approved these changes Nov 7, 2022

View reviewed changes

change: try using os path

864a90a

mufaddal-rohawala self-assigned this Nov 16, 2022

mufaddal-rohawala approved these changes Nov 16, 2022

View reviewed changes

src/sagemaker/spark/processing.py Show resolved Hide resolved

mufaddal-rohawala requested changes Nov 16, 2022

View reviewed changes

vikas0203 requested review from a team, trajanikant and claytonparnell and removed request for a team November 23, 2022 20:22

vikas0203 assigned trajanikant Nov 23, 2022

fix: unrelated test suite pollution, and update spark base class

8d7f82e

change: use copy library and re-spawn tests

a7a8c38

brockwade633 requested a review from a team as a code owner December 5, 2022 19:25

qidewenwhen approved these changes Dec 5, 2022

View reviewed changes

knikure reviewed Dec 5, 2022

View reviewed changes

knikure changed the title ~~bug-fix: support idempotency for framework and spark processors~~ fix: support idempotency for framework and spark processors Dec 5, 2022

knikure merged commit b68bcd9 into aws:master Dec 5, 2022

mizanfiu pushed a commit to mizanfiu/sagemaker-python-sdk that referenced this pull request Dec 13, 2022

fix: support idempotency for framework and spark processors (aws#3460)

e8f0753

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

mufaddal-rohawala added a commit that referenced this pull request Dec 20, 2022

fix: support idempotency for framework and spark processors (#3460)

f27f975

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

fix: support idempotency for framework and spark processors (aws#3460)

4617686

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

fix: support idempotency for framework and spark processors (aws#3460)

f11ba6a

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

fix: support idempotency for framework and spark processors (aws#3460)

969485c

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

fix: support idempotency for framework and spark processors (aws#3460)

6b7cf17

Co-authored-by: Brock Wade <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]>

fix: support idempotency for framework and spark processors #3460

fix: support idempotency for framework and spark processors #3460

Conversation

brockwade633 commented Nov 7, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 7, 2022

AWS CodeBuild CI Report

codecov-commenter commented Nov 10, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Nov 10, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 10, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 10, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 10, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 10, 2022

AWS CodeBuild CI Report

mufaddal-rohawala left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

mufaddal-rohawala commented Dec 1, 2022

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

knikure left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 5, 2022

AWS CodeBuild CI Report

brockwade633 commented Nov 7, 2022 •

edited

Loading

codecov-commenter commented Nov 10, 2022 •

edited

Loading