Skip to content

deprecation: remove explicit partitions key requirement on pysdk side. #4057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 5, 2023
Merged

deprecation: remove explicit partitions key requirement on pysdk side. #4057

merged 1 commit into from
Dec 5, 2023

Conversation

viclzhu
Copy link
Collaborator

@viclzhu viclzhu commented Aug 7, 2023 •
edited
Loading

Description of changes:
Clean up API for SMP v2.
partitions key will still be checked on library side for SMP v1.

Remove related test checking that error is thrown if partitions is not passed.

Testing done:
Ran tox -e py310 -- -s -vv tests/unit/test_fw_utils.py which is passing.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@viclzhu viclzhu requested a review from a team as a code owner August 7, 2023 23:39
@viclzhu viclzhu requested review from akrishna1995 and removed request for a team August 7, 2023 23:39
@viclzhu viclzhu changed the title deprecation: remove partitions key explicit requirement on pysdk side. deprecation: remove explicit partitions key requirement on pysdk side. Aug 7, 2023
@viclzhu viclzhu force-pushed the remove-partitions-check branch 2 times, most recently from 5f9d8b4 to adcc0c2 Compare August 7, 2023 23:42
@viclzhu
Copy link
Collaborator Author

viclzhu commented Aug 7, 2023

cc: @rahul003

@akrishna1995 akrishna1995 self-assigned this Aug 9, 2023
akrishna1995
akrishna1995 reviewed Aug 9, 2023
View reviewed changes
Copy link
Contributor

@akrishna1995 akrishna1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/bot run all

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: adcc0c2
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: adcc0c2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: adcc0c2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: adcc0c2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: adcc0c2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@codecov-commenter
Copy link

codecov-commenter commented Aug 9, 2023 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (57fd632) 86.72% compared to head (3f82852) 90.32%.

❗ Current head 3f82852 differs from pull request most recent head bd4161b. Consider uploading reports for the commit bd4161b to get more accurate results

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4057      +/-   ##
==========================================
+ Coverage   86.72%   90.32%   +3.59%     
==========================================
  Files        1197      320     -877     
  Lines      105633    29784   -75849     
==========================================
- Hits        91609    26902   -64707     
+ Misses      14024     2882   -11142

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

akrishna1995
akrishna1995 approved these changes Aug 14, 2023
View reviewed changes
Copy link
Contributor

@akrishna1995 akrishna1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Known flaky test failed: /amazon-sagemaker-examples/sagemaker-python-sdk/tensorflow_moving_from_framework_mode_to_script_mode/tensorflow_moving_from_framework_mode_to_script_mode.ipynb

LGTM

@akrishna1995 akrishna1995 force-pushed the remove-partitions-check branch from adcc0c2 to 693dfe2 Compare August 14, 2023 19:48
akrishna1995
akrishna1995 reviewed Aug 14, 2023
View reviewed changes
Copy link
Contributor

@akrishna1995 akrishna1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/bot run all

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 693dfe2
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 693dfe2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 693dfe2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 693dfe2
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 693dfe2
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@akrishna1995
Copy link
Contributor

Can you please look into the failing tests and confirm they are unrelated to your changes? Please run them locally

@akrishna1995
Copy link
Contributor

gentle ping on this. Can you please look into the failing tests and confirm they are unrelated to your changes? Please run them locally

@viclzhu viclzhu force-pushed the remove-partitions-check branch from 693dfe2 to c5b594b Compare September 11, 2023 19:37
@viclzhu
Copy link
Collaborator Author

viclzhu commented Sep 11, 2023

Ok yes, for sagemaker-python-sdk-pr, I ran tox -e py310 -- -s -vv tests/integ/test_huggingface_torch_distributed.py which was failing on the PR due to capacity issues. It is passing the test when I run it locally.

For sagemaker-python-sdk-slow-tests, the logs aren't clear what is failing though, not sure what I should run here.

@viclzhu
Copy link
Collaborator Author

viclzhu commented Sep 25, 2023

@akrishna1995 is there any further follow-up required on our side?

akrishna1995
akrishna1995 reviewed Oct 2, 2023
View reviewed changes
Copy link
Contributor

@akrishna1995 akrishna1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/bot run all

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: c5b594b
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: c5b594b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: c5b594b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: c5b594b
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: c5b594b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@goelakash goelakash force-pushed the remove-partitions-check branch from c5b594b to 43ae515 Compare October 9, 2023 23:49
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 43ae515
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 43ae515
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 43ae515
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 43ae515
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 43ae515
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@viclzhu viclzhu force-pushed the remove-partitions-check branch from 43ae515 to 3f82852 Compare November 20, 2023 19:43
@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 3f82852
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 3f82852
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 3f82852
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 3f82852
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 3f82852
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@viclzhu
deprecation: Remove explicit partitions key requirement on pysdk side.
bd4161b 
Clean up API for SMP v2.
`partitions` key will still be checked on library side for SMP v1.

Remove related test checking that error is thrown if partitions is not passed.
@viclzhu viclzhu force-pushed the remove-partitions-check branch from 3f82852 to bd4161b Compare December 4, 2023 18:47
@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: bd4161b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: bd4161b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: bd4161b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: bd4161b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mufaddal-rohawala
Copy link
Member

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: bd4161b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@knikure knikure merged commit fcc2e74 into aws:master Dec 5, 2023
@viclzhu viclzhu deleted the remove-partitions-check branch December 6, 2023 00:03
knikure pushed a commit to knikure/sagemaker-python-sdk that referenced this pull request May 8, 2024
@viclzhu @sagemaker-bot
deprecation: remove explicit partitions key requirement on pysdk si…
aa99b5b 
…de. (aws#4057)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akrishna1995 akrishna1995 akrishna1995 approved these changes

@rahul003 rahul003 rahul003 approved these changes

@SanjanaKallol SanjanaKallol SanjanaKallol approved these changes

Assignees

@akrishna1995 akrishna1995

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@viclzhu @sagemaker-bot @codecov-commenter @akrishna1995 @mufaddal-rohawala @rahul003 @SanjanaKallol @knikure