fix:ResourceConflictException from AWS Lambda on pipeline upsert #3106

nmadan · 2022-05-12T20:05:38Z

Issue #, if available:

Description of changes: PR 3052 introduced a bug where pipelines that were upserted would try to update user's lambda step function code twice. This resulted in the following error for the second update_function_code call:

[ERROR]	2022-05-12T03:49:04.962Z	c0ba672c-1dbd-4894-ab57-ec1ecfc6c448	Encountered exception {'Message': 'The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:<region>:<account-id>:function:<function-name>', 'Code': 'ResourceConflictException'}

This PR aims to fix this issue but changing the pipeline upsert logic to either create or update pipeline (not both).
It also includes retries in case lambda does throw a ResourceConflictException

Testing done: unit tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-05-12T20:27:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a44939a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T20:42:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a44939a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-05-12T20:42:10Z

Codecov Report

Merging #3106 (a44939a) into master (92b1d47) will increase coverage by 0.03%.
The diff coverage is 72.00%.

@@            Coverage Diff             @@
##           master    #3106      +/-   ##
==========================================
+ Coverage   89.51%   89.55%   +0.03%     
==========================================
  Files         199      199              
  Lines       17177    17181       +4     
==========================================
+ Hits        15376    15386      +10     
+ Misses       1801     1795       -6

Impacted Files	Coverage Δ
src/sagemaker/workflow/pipeline.py	`90.19% <46.15%> (+2.03%)`	⬆️
src/sagemaker/lambda_helper.py	`97.24% <100.00%> (+2.90%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92b1d47...a44939a. Read the comment docs.

sagemaker-bot · 2022-05-12T20:53:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a44939a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T21:10:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a44939a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T21:22:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a44939a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T21:24:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 8199447
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T21:40:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8199447
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T21:53:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 8199447
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T22:02:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8199447
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T22:25:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 8199447
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-12T23:19:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 8199447
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2022-05-13T00:52:15Z

src/sagemaker/workflow/pipeline.py

        try:
+            self.describe()
+        except ClientError:
+            exists = False


nit: is there a better way to check this?

ListPipelines is another option but I think Describe works better since List calls are paginated.

Even when using describe, excepting ClientError to mean it does not exist, there can be some other cause of ClientError right, since it's top level API error

Fixed in latest revision

sagemaker-bot · 2022-05-13T21:17:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 26a0278
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T21:52:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 222937c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:01:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 26a0278
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:10:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 222937c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:16:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 26a0278
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:18:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 222937c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:20:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 26a0278
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:24:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 222937c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/workflow/pipeline.py

sagemaker-bot · 2022-05-13T22:31:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7b89814
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T22:35:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 222937c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T23:16:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7b89814
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T23:17:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 7b89814
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T23:23:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 7b89814
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-13T23:32:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 7b89814
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T18:42:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6cbab48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T19:05:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6cbab48
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T19:08:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6cbab48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T19:19:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6cbab48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T19:27:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6cbab48
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T21:35:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6cbab48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-16T22:08:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6cbab48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

staubhp approved these changes May 12, 2022

View reviewed changes

mufaddal-rohawala reviewed May 13, 2022

View reviewed changes

nmadan commented May 13, 2022

View reviewed changes

src/sagemaker/workflow/pipeline.py Show resolved Hide resolved

fix:ResourceConflictException from AWS lambda on pipeline upsert

6cbab48

mufaddal-rohawala approved these changes May 17, 2022

View reviewed changes

shreyapandit approved these changes May 17, 2022

View reviewed changes

shreyapandit changed the title ~~fix:ResourceConflictException from AWS lambda on pipeline upsert~~ fix:ResourceConflictException from AWS Lambda on pipeline upsert May 17, 2022

shreyapandit merged commit 6f2cd5a into aws:master May 17, 2022

fix:ResourceConflictException from AWS Lambda on pipeline upsert #3106

fix:ResourceConflictException from AWS Lambda on pipeline upsert #3106

Conversation

nmadan commented May 12, 2022

Merge Checklist

General

Tests

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

codecov-commenter commented May 12, 2022 • edited Loading

Codecov Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 12, 2022

AWS CodeBuild CI Report

mufaddal-rohawala May 13, 2022

Choose a reason for hiding this comment

nmadan May 13, 2022

Choose a reason for hiding this comment

mufaddal-rohawala May 13, 2022

Choose a reason for hiding this comment

nmadan May 13, 2022

Choose a reason for hiding this comment

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 16, 2022

codecov-commenter commented May 12, 2022 •

edited

Loading