feature: add joinsource to DataConfig #2718

keerthanvasist · 2021-10-20T06:48:22Z

Description of changes:
Add joinsource to DataConfig.

Joinsource parameter in DataConfig will represent the name of a column in the dataset that acts as an identifier (for instance, while performing a join), and not used for any other computations.

Testing done:
All tests have passed. I have updated unit tests to include joinsource as well.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-10-20T07:01:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a49c487
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T07:33:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a49c487
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-20T07:43:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a49c487
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

karan6181

LGTM. Thanks!

xgchena · 2021-10-20T20:57:59Z

General comments to the PR description,

Please add a brief introduction to the feature you are adding.
There is a "Merge Checklist", please review it and acknowledge that you followed the instructions.

xgchena · 2021-10-20T21:07:59Z

src/sagemaker/clarify.py

@@ -34,6 +34,7 @@ def __init__(
        label=None,
        headers=None,
        features=None,
+        joinsource=None,


Red flag: for backward-compatibility, new parameters should be put to the end.

Snake style naming like "join_source"?

Updated arguments order.
joinsource is consistent with how it is used in other places like analysis_config. So I think we should keep it.

sagemaker-bot · 2021-10-21T06:40:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3d703c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T07:14:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 3d703c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T07:22:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3d703c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/clarify.py

sagemaker-bot · 2021-10-21T16:39:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6b055d6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T17:12:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6b055d6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T17:21:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6b055d6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T18:50:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c21cb86
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jeniyat

LGTM

sagemaker-bot · 2021-10-21T19:23:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c21cb86
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T19:30:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c21cb86
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T22:13:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: be95fee
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T22:57:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: be95fee
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-21T22:58:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: be95fee
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

xgchena

The change looks good. Since now it doesn't include a integ test, if you still have the test cases locally my suggestion is that you run them against your indivisual account to validate the E2E workflow.

sagemaker-bot · 2021-10-22T08:02:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1399bdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-22T08:46:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1399bdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ahsan-z-khan · 2021-10-22T19:59:33Z

@keerthanvasist have you done the test @xgchena suggested?

keerthanvasist · 2021-10-22T20:06:45Z

@keerthanvasist have you done the test @xgchena suggested?

@ahsan-z-khan I am looking into it. I will post here once I am done.

navinsoni

Waiting on Container changes.

keerthanvasist · 2021-10-29T17:59:19Z

This change is manually E2E tested, and ready to go.

sagemaker-bot · 2021-10-29T21:19:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3a699ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-29T22:01:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3a699ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-29T22:08:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1c3a2ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-29T22:49:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1c3a2ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-29T23:05:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 1c3a2ad
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-30T00:30:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 1c3a2ad
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

karan6181 previously approved these changes Oct 20, 2021

View reviewed changes

xgchena suggested changes Oct 20, 2021

View reviewed changes

keerthanvasist dismissed karan6181’s stale review via 3d703c4 October 21, 2021 06:26

keerthanvasist force-pushed the joinsource branch from a49c487 to 3d703c4 Compare October 21, 2021 06:26

navinsoni reviewed Oct 21, 2021

View reviewed changes

src/sagemaker/clarify.py Outdated Show resolved Hide resolved

ahsan-z-khan previously approved these changes Oct 21, 2021

View reviewed changes

jeniyat previously approved these changes Oct 21, 2021

View reviewed changes

shreyapandit previously approved these changes Oct 21, 2021

View reviewed changes

xgchena previously approved these changes Oct 21, 2021

View reviewed changes

navinsoni suggested changes Oct 25, 2021

View reviewed changes

mufaddal-rohawala previously approved these changes Oct 26, 2021

View reviewed changes

keerthanvasist dismissed stale reviews from mufaddal-rohawala, xgchena, shreyapandit, jeniyat, and ahsan-z-khan via 3a699ab October 29, 2021 21:04

keerthanvasist force-pushed the joinsource branch from 1399bdb to 3a699ab Compare October 29, 2021 21:04

feature: add joinsource to DataConfig

1c3a2ad

keerthanvasist force-pushed the joinsource branch from 3a699ab to 1c3a2ad Compare October 29, 2021 21:54

navinsoni approved these changes Oct 29, 2021

View reviewed changes

shreyapandit approved these changes Oct 30, 2021

View reviewed changes

navinsoni merged commit 9be4c8a into aws:master Oct 30, 2021

EthanShouhanCheng pushed a commit to SissiChenxy/sagemaker-python-sdk that referenced this pull request Jan 11, 2022

feature: add joinsource to DataConfig (aws#2718)

14c163a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: add joinsource to DataConfig #2718

feature: add joinsource to DataConfig #2718

keerthanvasist commented Oct 20, 2021 •

edited

Loading

sagemaker-bot commented Oct 20, 2021

sagemaker-bot commented Oct 20, 2021

sagemaker-bot commented Oct 20, 2021

karan6181 left a comment

xgchena commented Oct 20, 2021

xgchena Oct 20, 2021

xgchena Oct 20, 2021

keerthanvasist Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

jeniyat left a comment

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

sagemaker-bot commented Oct 21, 2021

xgchena left a comment

sagemaker-bot commented Oct 22, 2021

sagemaker-bot commented Oct 22, 2021

ahsan-z-khan commented Oct 22, 2021

keerthanvasist commented Oct 22, 2021

navinsoni left a comment

keerthanvasist commented Oct 29, 2021 •

edited

Loading

sagemaker-bot commented Oct 29, 2021

sagemaker-bot commented Oct 29, 2021

sagemaker-bot commented Oct 29, 2021

sagemaker-bot commented Oct 29, 2021

sagemaker-bot commented Oct 29, 2021

sagemaker-bot commented Oct 30, 2021

feature: add joinsource to DataConfig #2718

feature: add joinsource to DataConfig #2718

Conversation

keerthanvasist commented Oct 20, 2021 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 20, 2021

AWS CodeBuild CI Report

karan6181 left a comment

Choose a reason for hiding this comment

xgchena commented Oct 20, 2021

xgchena Oct 20, 2021

Choose a reason for hiding this comment

xgchena Oct 20, 2021

Choose a reason for hiding this comment

keerthanvasist Oct 21, 2021

Choose a reason for hiding this comment

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

jeniyat left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 21, 2021

AWS CodeBuild CI Report

xgchena left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Oct 22, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 22, 2021

AWS CodeBuild CI Report

ahsan-z-khan commented Oct 22, 2021

keerthanvasist commented Oct 22, 2021

navinsoni left a comment

Choose a reason for hiding this comment

keerthanvasist commented Oct 29, 2021 • edited Loading

sagemaker-bot commented Oct 29, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 29, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 29, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 29, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 29, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 30, 2021

AWS CodeBuild CI Report

keerthanvasist commented Oct 20, 2021 •

edited

Loading

keerthanvasist commented Oct 29, 2021 •

edited

Loading