feature: Feature Store dataset builder, delete_record, get_record, list_feature_group #3534

mizanfiu · 2022-12-13T09:06:42Z

Issue #, if available:
Merging feature store commits from https://github.com/aws/sagemaker-python-sdk-staging
Feature Store dataset builder, delete_record, get_record, list_feature_group
Description of changes:
Feature Store dataset builder, delete_record, get_record, list_feature_group
Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Co-authored-by: Eric Zou <[email protected]>

) * Bug fixed - as_of, event_range, join, default behavior and duplicates and tests Bugs: 1. as_of was not working properly on deleted events 2. Same event_time_range 3. Join was not working when including feature names 4. Default sql was returning only most recent, whereas it should all excluding duplicates 5. Include duplicates was not return all non-deleted data 6. instanceof(dataframe) case was also applied to non-df cases while join 7. Include column was returning unnecessary columns. * Fix on pylint error * Fix on include_duplicated_records for panda data frames * Fix format issue for black * Bug fixed related to line break * Bug fix related to dataframe and inclde_deleted_record and include_duplicated_record * Addressed comments and code refactored * changed to_csv to to_csv_file and added error messages for query limit and recent record limit * Revert a change which was not intended * Resolved the leak of feature group deletion in integration test

navinsoni

/bot run all

sagemaker-bot · 2022-12-13T18:42:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: abab2a1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T18:44:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: abab2a1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T19:20:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: abab2a1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T19:21:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: abab2a1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T19:43:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: abab2a1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

/bot run all

mizanfiu · 2022-12-13T20:11:12Z

/bot run all

sagemaker-bot · 2022-12-13T20:33:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9f3b4b6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T20:39:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 9f3b4b6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T20:40:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9f3b4b6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-12-13T21:02:12Z

Codecov Report

Merging #3534 (9f3b4b6) into master (1fa2377) will decrease coverage by 0.32%.
The diff coverage is 87.51%.

@@            Coverage Diff             @@
##           master    #3534      +/-   ##
==========================================
- Coverage   89.17%   88.84%   -0.33%     
==========================================
  Files         204      215      +11     
  Lines       18979    20925    +1946     
==========================================
+ Hits        16924    18591    +1667     
- Misses       2055     2334     +279

Impacted Files	Coverage Δ
src/sagemaker/huggingface/model.py	`89.87% <ø> (ø)`
src/sagemaker/pipeline.py	`85.33% <ø> (ø)`
src/sagemaker/sklearn/model.py	`92.98% <ø> (-1.76%)`	⬇️
src/sagemaker/sparkml/model.py	`100.00% <ø> (ø)`
...sagemaker/workflow/monitor_batch_transform_step.py	`0.00% <0.00%> (ø)`
src/sagemaker/xgboost/model.py	`96.36% <ø> (-1.82%)`	⬇️
...agemaker/model_monitor/clarify_model_monitoring.py	`89.84% <27.58%> (-5.09%)`	⬇️
src/sagemaker/transformer.py	`85.41% <40.62%> (-9.07%)`	⬇️
.../sagemaker/huggingface/training_compiler/config.py	`90.47% <50.00%> (ø)`
src/sagemaker/jumpstart/cache.py	`94.36% <60.00%> (-1.32%)`	⬇️
... and 59 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-12-13T21:02:17Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9f3b4b6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-13T22:48:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 9f3b4b6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…st_feature_group (aws#3534) Co-authored-by: Eric Zou <[email protected]> Co-authored-by: Yiming Zou <[email protected]> Co-authored-by: Brandon Chatham <[email protected]> Co-authored-by: jiapinw <[email protected]>

…st_feature_group (#3534) Co-authored-by: Eric Zou <[email protected]> Co-authored-by: Yiming Zou <[email protected]> Co-authored-by: Brandon Chatham <[email protected]> Co-authored-by: jiapinw <[email protected]>

…st_feature_group (aws#3534) Co-authored-by: Eric Zou <[email protected]> Co-authored-by: Yiming Zou <[email protected]> Co-authored-by: Brandon Chatham <[email protected]> Co-authored-by: jiapinw <[email protected]>

imingtsou and others added 15 commits December 13, 2022 01:00

Add list_feature_groups API (aws#647)

ad9e259

feat: Feature/get record api (aws#650)

031ba15

Co-authored-by: Eric Zou <[email protected]>

Add delete_record API (aws#664)

8c5a8c9

feat: Add DatasetBuilder class (aws#667)

699f9b0

Co-authored-by: Eric Zou <[email protected]>

feat: Add to_csv method in DatasetBuilder (aws#699)

9fb4ae2

feat: Add pandas.Dataframe as base case (aws#708)

b0e537b

feat: Add with_feature_group method in DatasetBuilder (aws#726)

b01c3ae

feat: Handle merge and timestamp filters (aws#727)

c06ed9b

feat: Add to_dataframe method in DatasetBuilder (aws#729)

93a867c

Address TODOs (aws#731)

98b60c0

Unit test for DatasetBuilder (aws#734)

66e18a4

fix: Fix list_feature_groups max_results (aws#744)

7723c65

Add integration tests for create_dataset (aws#743)

13c4d5b

feature: Aggregate commits

a888bd5

mizanfiu requested a review from a team as a code owner December 13, 2022 09:06

mizanfiu requested review from claytonparnell and removed request for a team December 13, 2022 09:06

navinsoni reviewed Dec 13, 2022

View reviewed changes

navinsoni approved these changes Dec 13, 2022

View reviewed changes

Merge branch 'aws:master' into master

9f3b4b6

navinsoni reviewed Dec 13, 2022

View reviewed changes

navinsoni changed the title ~~Feature Store dataset builder, delete_record, get_record, list_feature_group~~ feature: Feature Store dataset builder, delete_record, get_record, list_feature_group Dec 14, 2022

navinsoni merged commit bd96ec5 into aws:master Dec 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Feature Store dataset builder, delete_record, get_record, list_feature_group #3534

feature: Feature Store dataset builder, delete_record, get_record, list_feature_group #3534

mizanfiu commented Dec 13, 2022 •

edited

Loading

navinsoni left a comment

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

navinsoni left a comment

mizanfiu commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

codecov-commenter commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

feature: Feature Store dataset builder, delete_record, get_record, list_feature_group #3534

feature: Feature Store dataset builder, delete_record, get_record, list_feature_group #3534

Conversation

mizanfiu commented Dec 13, 2022 • edited Loading

Merge Checklist

General

Tests

navinsoni left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

navinsoni left a comment

Choose a reason for hiding this comment

mizanfiu commented Dec 13, 2022

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

codecov-commenter commented Dec 13, 2022

Codecov Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 13, 2022

AWS CodeBuild CI Report

mizanfiu commented Dec 13, 2022 •

edited

Loading