feature: support JsonGet for all step types #2506

jerrypeng7773 · 2021-07-01T23:52:29Z

Issue #, if available:
#2426

Description of changes:

deprecate the JsonGet in condition_step.py
add JsonGet support to all type of steps in functions.py

Testing done:

unit and integ tests.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…rs to sagemaker.wrangler.ingestion

sagemaker-bot · 2021-07-02T00:03:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: db83c4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-02T00:30:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: db83c4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-02T00:35:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: db83c4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-02T00:38:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: db83c4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-02T01:43:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: db83c4b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-22T04:09:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 989875e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-22T04:35:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 989875e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-22T04:36:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 989875e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-22T04:44:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 989875e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-22T06:23:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 989875e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-23T16:19:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 10d0fc7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-23T16:46:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 10d0fc7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-23T16:48:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 10d0fc7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-23T16:54:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 10d0fc7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-23T17:38:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 10d0fc7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-28T19:22:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: fcd2615
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-28T19:49:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: fcd2615
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-28T19:56:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: fcd2615
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-28T20:01:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: fcd2615
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-07-28T20:17:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: fcd2615
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-02T17:22:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d3f5761
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

shreyapandit · 2021-08-02T17:26:39Z

src/sagemaker/workflow/functions.py

@@ -54,3 +55,36 @@ def expr(self):
                ],
            },
        }
+
+
+@attr.s


For this class, we are introducing the same duplicated block of code here as in condition_step.py, while marking the block there as deprecated. The class itself is the same. If we still plan to reuse the same code in condition_step as well as here, why can't we pull this out somewhere common, like utils, where both can import it

sagemaker-bot · 2021-08-02T17:53:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d3f5761
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-02T17:56:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d3f5761
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-02T18:04:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d3f5761
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-08-02T19:58:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d3f5761
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jerrypeng7773 and others added 23 commits May 10, 2021 20:18

add helper function to generate no-op (data ingestion only) recipe

868db55

Merge branch 'aws:master' into master

21bedbb

separate flow generation by source input type + move generation helpe…

854dd10

…rs to sagemaker.wrangler.ingestion

Merge branch 'aws:master' into master

8798b65

create an internal helper function to generate output node

69ae4bd

Merge branch 'master' of github.com:jerrypeng7773/sagemaker-python-sdk

a6a8449

Merge branch 'aws:master' into master

2aa256e

add ingestion test using dw processor via pipeline execution

06557a8

Merge branch 'aws:master' into master

dcbfd13

verify the fg query df

fc6522e

Merge branch 'master' into master

b6f9371

fix tests

86fa47d

Merge branch 'master' into master

05ccfa6

Merge branch 'aws:master' into master

0716e9f

add tuning step support

7ca5af4

fix docstyle check

8cf18b8

add helper function to get tuning step top performing model s3 uri

1f95b82

Merge branch 'aws:master' into master

1b9d66b

Merge branch 'aws:master' into master

603b934

Merge branch 'master' of github.com:jerrypeng7773/sagemaker-python-sdk

664f2a8

Merge branch 'master' into master

a8755ec

Merge branch 'aws:master' into master

e25d36c

make JsonGet compatible to all steps and put it into functions.py

db83c4b

jerrypeng7773 marked this pull request as ready for review July 2, 2021 17:23

leoroy approved these changes Jul 2, 2021

View reviewed changes

Merge branch 'master' into fix-issue-2426

989875e

Merge branch 'master' into fix-issue-2426

10d0fc7

Merge branch 'master' into fix-issue-2426

fcd2615

ahsan-z-khan added the component: pipelines Relates to the SageMaker Pipeline Platform label Jul 28, 2021

Merge branch 'master' into fix-issue-2426

d3f5761

shreyapandit reviewed Aug 2, 2021

View reviewed changes

ahsan-z-khan approved these changes Aug 2, 2021

View reviewed changes

ahsan-z-khan changed the title ~~support JsonGet for all step types.~~ feature: support JsonGet for all step types Aug 2, 2021

ahsan-z-khan merged commit 5c8ef31 into aws:master Aug 2, 2021

jkroll-aws mentioned this pull request Oct 21, 2021

doc: Add JsonGet v2 deprecation #2720

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: support JsonGet for all step types #2506

feature: support JsonGet for all step types #2506

jerrypeng7773 commented Jul 1, 2021 •

edited by ahsan-z-khan

Loading

sagemaker-bot commented Jul 2, 2021

sagemaker-bot commented Jul 2, 2021

sagemaker-bot commented Jul 2, 2021

sagemaker-bot commented Jul 2, 2021

sagemaker-bot commented Jul 2, 2021

sagemaker-bot commented Jul 22, 2021

sagemaker-bot commented Jul 22, 2021

sagemaker-bot commented Jul 22, 2021

sagemaker-bot commented Jul 22, 2021

sagemaker-bot commented Jul 22, 2021

sagemaker-bot commented Jul 23, 2021

sagemaker-bot commented Jul 23, 2021

sagemaker-bot commented Jul 23, 2021

sagemaker-bot commented Jul 23, 2021

sagemaker-bot commented Jul 23, 2021

sagemaker-bot commented Jul 28, 2021

sagemaker-bot commented Jul 28, 2021

sagemaker-bot commented Jul 28, 2021

sagemaker-bot commented Jul 28, 2021

sagemaker-bot commented Jul 28, 2021

sagemaker-bot commented Aug 2, 2021

shreyapandit Aug 2, 2021

sagemaker-bot commented Aug 2, 2021

sagemaker-bot commented Aug 2, 2021

sagemaker-bot commented Aug 2, 2021

sagemaker-bot commented Aug 2, 2021

@@ @@ -54,3 +55,36 @@ def expr(self): @@
                               ],
                           },
                       }
+              @attr.s

feature: support JsonGet for all step types #2506

feature: support JsonGet for all step types #2506

Conversation

jerrypeng7773 commented Jul 1, 2021 • edited by ahsan-z-khan Loading

Merge Checklist

General

Tests

sagemaker-bot commented Jul 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 22, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 23, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2021

AWS CodeBuild CI Report

shreyapandit Aug 2, 2021

Choose a reason for hiding this comment

sagemaker-bot commented Aug 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2021

AWS CodeBuild CI Report

jerrypeng7773 commented Jul 1, 2021 •

edited by ahsan-z-khan

Loading