Feature/new fg utils #3476

JoseJuan98 · 2022-11-18T15:03:41Z

Issue #, if available:

Description of changes: added a few methods to automate processes to work with Feature Groups, one to prepare a FG before creation and another to extract directly a dataframe from a FG faster.
Also, I extended the method FeatureGroup.as_dataframe() to accept kwargs to specify some parameters in the pandas.read_csv use to create the dataframe as I encounter some problems with encoding and dtypes that could be fix specifying low_memory=False and the encoding used.

Testing done:
Local tests with tox following the contribution guide.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

navinsoni

@JoseJuan98 Can you please update description with what's happening in this PR

JoseJuan98 · 2022-11-23T21:49:29Z

@JoseJuan98 Can you please update description with what's happening in this PR

I updated it, please let me know if something is not clear :)

navinsoni

/bot run all

sagemaker-bot · 2022-12-02T20:25:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 64b6fa9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-02T20:26:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 64b6fa9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-02T20:27:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 64b6fa9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

@JoseJuan98 Can you please address unit-test and other test failures.
The failure are related to styling. you can run following command to locally to fix it

tox -e flake8,pylint,docstyle,black-check,twine

sagemaker-bot · 2022-12-02T21:28:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 64b6fa9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-02T22:18:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 64b6fa9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

JoseJuan98 · 2022-12-03T00:35:28Z

@JoseJuan98 Can you please address unit-test and other test failures. The failure are related to styling. you can run following command to locally to fix it

tox -e flake8,pylint,docstyle,black-check,twine

@navinsoni I made all changes in the recents pushes, now all should be fixed

______________________________________________________ summary _______________________________________________________
  flake8: commands succeeded
  pylint: commands succeeded
  docstyle: commands succeeded
  black-check: commands succeeded
  twine: commands succeeded
  congratulations :)

navinsoni · 2022-12-15T22:47:17Z

@JoseJuan98 Can you please fix conflicts.

JoseJuan98 · 2022-12-16T10:23:02Z

@JoseJuan98 Can you please fix conflicts.

@navinsoni merged, conflicts solved and the tox tests passed

JoseJuan98 · 2022-12-20T11:33:09Z

@JoseJuan98 Can you please fix conflicts.

@navinsoni merged and solved conflicts with the most recent master

jpenagom

Reviewed code style and docstyle

codecov-commenter · 2023-01-04T01:51:43Z

Codecov Report

Merging #3476 (8fbc1c6) into master (1fa2377) will increase coverage by 0.38%.
The diff coverage is n/a.

❗ Current head 8fbc1c6 differs from pull request most recent head b31a0fe. Consider uploading reports for the commit b31a0fe to get more accurate results

@@            Coverage Diff             @@
##           master    #3476      +/-   ##
==========================================
+ Coverage   89.17%   89.55%   +0.38%     
==========================================
  Files         204      960     +756     
  Lines       18979    88796   +69817     
==========================================
+ Hits        16924    79523   +62599     
- Misses       2055     9273    +7218

Impacted Files	Coverage Δ
src/sagemaker/algorithm.py
src/sagemaker/amazon/amazon_estimator.py
src/sagemaker/amazon/factorization_machines.py
src/sagemaker/amazon/ipinsights.py
src/sagemaker/amazon/kmeans.py
src/sagemaker/amazon/knn.py
src/sagemaker/amazon/lda.py
src/sagemaker/amazon/linear_learner.py
src/sagemaker/amazon/ntm.py
src/sagemaker/amazon/object2vec.py
... and 1154 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

jpenagom · 2023-01-25T17:46:51Z

Hi @navinsoni & @trajanikant, could you please give me more information about the status of this PR? Is there anything else I could do to resolve the changes requested by you?

The sphinx build was failing, I have fixed it, now all tox tests environments suggested by you and the documentation are passing.

Thank you on advance :)

trajanikant

/bot run all

sagemaker-bot · 2023-02-08T20:03:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 85af929
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* build: reset soft * docs(docs): update docs with USE_SHORT_LIVED_CREDENTIALS bullet * style(format): fix rst code format

* updates docstrings for datasetbuilder members * feature store user guide updates * update to chain code snippet, updates api ref link * update description to be before code * removes trailing whitespace * remove whitespace from user guide

Co-authored-by: Raymond Liu <[email protected]>

…re/new_fg_utils

…re/new_fg_utils # Conflicts: # .pylintrc # CHANGELOG.md # VERSION # src/sagemaker/feature_group_utils.py # src/sagemaker/feature_store/feature_group.py # src/sagemaker/image_uri_config/tensorflow.json # src/sagemaker/session.py # src/sagemaker/tuner.py # src/sagemaker/utils.py # tests/conftest.py # tests/integ/test_feature_store.py # tests/unit/sagemaker/feature_store/test_feature_store.py

JoseJuan98 · 2023-03-04T13:17:10Z

@JoseJuan98 Can you please rebase these changes on master branch instead of merging. It is showing 42 files in diff

Hi @navinsoni , I tried to rebase my changes to master but anyways it rejected the push after the rebase and asked me to merge the branches before pushing, so it didn't solve it.

The only file affected by my changes are:

src/sagemaker/feature_group_utils.py
src/sagemaker/utils.py
src/sagemaker/feature_store/feature_group.py
test/unit/sagemaker/feature_store/test_feature_group_utils.py
test/unit/sagemaker/feature_store/test_feature_group.py
test/integ/test_feature_store.py

The rest of the file were added by the merge file. If it doesn't work I am afraid I will have to start a new PR by rebasing to master

navinsoni · 2023-03-08T20:51:15Z

@JoseJuan98 you will have to rebase the changes and force push it on your repo. It will be error prone to review this PR without that change.

JoseJuan98 · 2023-03-09T11:49:09Z

@JoseJuan98 you will have to rebase the changes and force push it on your repo. It will be error prone to review this PR without that change.

It has been very tiring and very error prompt to do that, so I decided to create a new PR using rebase, so now it only appears the new commits and the changes, please check link.

Shall I close this PR and continue from the other one?

navinsoni · 2023-03-09T17:20:39Z

@JoseJuan98 Thanks for creating new PR, closing this one.

vikas0203 requested review from a team, navinsoni and trajanikant and removed request for a team November 21, 2022 23:48

vikas0203 assigned navinsoni Nov 21, 2022

navinsoni reviewed Nov 23, 2022

View reviewed changes

navinsoni reviewed Dec 2, 2022

View reviewed changes

navinsoni suggested changes Dec 2, 2022

View reviewed changes

jpenagom requested a review from a team as a code owner December 2, 2022 23:05

mufaddal-rohawala force-pushed the master branch from bae3b8b to d000bdc Compare December 20, 2022 00:31

mufaddal-rohawala force-pushed the master branch 2 times, most recently from 8fbc1c6 to e6b8b15 Compare December 22, 2022 17:49

jpenagom reviewed Jan 2, 2023

View reviewed changes

navinsoni self-requested a review January 4, 2023 01:00

trajanikant reviewed Feb 8, 2023

View reviewed changes

trajanikant assigned trajanikant and unassigned navinsoni Feb 8, 2023

wcarpenter1-godaddy and others added 20 commits March 4, 2023 13:58

feature: allow use of short lived creds for local container (aws#3501)

1b798b0

* build: reset soft * docs(docs): update docs with USE_SHORT_LIVED_CREDENTIALS bullet * style(format): fix rst code format

update development version to v2.135.2.dev0

a682321

feature: Add DLC accounts for MEL Region (aws#3648)

5b7755f

prepare release v2.135.0

dfd2601

prepare release v2.135.1.post0

e59b068

update development version to v2.135.1.dev0

36d0f27

update development version to v2.135.2.dev0

084af58

feature: Decouple model.right_size() from model registry (aws#3688)

a275880

Co-authored-by: Raymond Liu <[email protected]>

change: Add djl 0.21.0 dlc images (aws#3685)

1cb2722

Merge remote-tracking branch 'origin/feature/new_fg_utils' into featu…

ddf5fc5

…re/new_fg_utils

Merge branch 'master' into feature/new_fg_utils

fbcd13b

Merge branch 'master' into feature/new_fg_utils

b04fc3f

Merge branch 'master' into feature/new_fg_utils

1d53a8a

Merge branch 'master' into feature/new_fg_utils

ad753ac

Merge branch 'master' into feature/new_fg_utils

6c413dd

Merge branch 'master' into feature/new_fg_utils

aa053f9

Merge branch 'master' into feature/new_fg_utils

9f6f704

Merge branch 'master' into feature/new_fg_utils

1313ee7

JoseJuan98 added 4 commits March 6, 2023 16:56

fix: linting and tox tests

429c179

fix: pylint

708fb7c

Merge branch 'master' into feature/new_fg_utils

50304ff

Merge branch 'master' into feature/new_fg_utils

b0db2b9

navinsoni closed this Mar 9, 2023

jpenagom deleted the feature/new_fg_utils branch March 10, 2023 08:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/new fg utils #3476

Feature/new fg utils #3476

JoseJuan98 commented Nov 18, 2022 •

edited

Loading

navinsoni left a comment

JoseJuan98 commented Nov 23, 2022

navinsoni left a comment

sagemaker-bot commented Dec 2, 2022

sagemaker-bot commented Dec 2, 2022

sagemaker-bot commented Dec 2, 2022

navinsoni left a comment

sagemaker-bot commented Dec 2, 2022

sagemaker-bot commented Dec 2, 2022

JoseJuan98 commented Dec 3, 2022

navinsoni commented Dec 15, 2022

JoseJuan98 commented Dec 16, 2022

JoseJuan98 commented Dec 20, 2022

jpenagom left a comment

codecov-commenter commented Jan 4, 2023 •

edited

Loading

jpenagom commented Jan 25, 2023 •

edited

Loading

trajanikant left a comment

sagemaker-bot commented Feb 8, 2023

JoseJuan98 commented Mar 4, 2023 •

edited

Loading

navinsoni commented Mar 8, 2023

JoseJuan98 commented Mar 9, 2023

navinsoni commented Mar 9, 2023

Feature/new fg utils #3476

Feature/new fg utils #3476

Conversation

JoseJuan98 commented Nov 18, 2022 • edited Loading

Merge Checklist

General

Tests

navinsoni left a comment

Choose a reason for hiding this comment

JoseJuan98 commented Nov 23, 2022

navinsoni left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

navinsoni left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Dec 2, 2022

AWS CodeBuild CI Report

JoseJuan98 commented Dec 3, 2022

navinsoni commented Dec 15, 2022

JoseJuan98 commented Dec 16, 2022

JoseJuan98 commented Dec 20, 2022

jpenagom left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jan 4, 2023 • edited Loading

Codecov Report

jpenagom commented Jan 25, 2023 • edited Loading

trajanikant left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Feb 8, 2023

AWS CodeBuild CI Report

JoseJuan98 commented Mar 4, 2023 • edited Loading

navinsoni commented Mar 8, 2023

JoseJuan98 commented Mar 9, 2023

navinsoni commented Mar 9, 2023

JoseJuan98 commented Nov 18, 2022 •

edited

Loading

codecov-commenter commented Jan 4, 2023 •

edited

Loading

jpenagom commented Jan 25, 2023 •

edited

Loading

JoseJuan98 commented Mar 4, 2023 •

edited

Loading