feature: Add PyTorch 1.13.1 to SDK #3587

ghost · 2023-01-12T17:33:20Z

Issue #, if available:

Description of changes:
Adding PyTorch 1.13.1 Training and Inference images to SageMaker SDK.

Testing done:
Unit tests have passed

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-01-12T17:35:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f34555b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-12T17:35:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f34555b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-12T17:35:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f34555b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-12T18:15:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f34555b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-01-12T18:19:59Z

Codecov Report

Merging #3587 (64bb0e8) into master (4d95b05) will decrease coverage by 0.79%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3587      +/-   ##
==========================================
- Coverage   89.59%   88.80%   -0.79%     
==========================================
  Files         968      227     -741     
  Lines       89600    22157   -67443     
==========================================
- Hits        80279    19677   -60602     
+ Misses       9321     2480    -6841

Impacted Files	Coverage Δ
src/sagemaker/fw_utils.py	`90.49% <ø> (ø)`
...3.8/site-packages/sagemaker/serverless/__init__.py
...3.10/site-packages/sagemaker/experiments/_utils.py
...ite-packages/sagemaker/jumpstart/notebook_utils.py
.../lib/python3.7/site-packages/sagemaker/pipeline.py
...python3.8/site-packages/sagemaker/xgboost/utils.py
...hon3.7/site-packages/sagemaker/serverless/model.py
.../site-packages/sagemaker/workflow/callback_step.py
...e-packages/sagemaker/cli/compatibility/__init__.py
...ackages/sagemaker/cli/compatibility/v2/__init__.py
... and 1186 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-01-12T18:26:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f34555b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chore: add PT-1.13.1 for tr

sagemaker-bot · 2023-01-23T23:39:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 67eb6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-23T23:40:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 67eb6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-23T23:43:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 67eb6b2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-23T23:59:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 67eb6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T01:51:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 67eb6b2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T15:04:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: bf203e3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T15:04:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: bf203e3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T15:04:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: bf203e3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T15:44:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: bf203e3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T15:56:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: bf203e3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T17:11:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9b25f3f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-24T17:23:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1a6ad3c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-25T21:45:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dcef8d4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-25T21:55:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: dcef8d4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knikure

/bot run local-mode-tests, pr, slow-tests

sagemaker-bot · 2023-01-25T22:05:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: dcef8d4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-25T22:11:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dcef8d4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kenny-ezirim · 2023-01-25T22:46:39Z

tests/conftest.py

        return "py3"
+    else:
+        return request.param


This is a significant change in default behavior of this function pytorch_training_py_version.

Insert the following code after line 225 in original code.
if Version(pytorch_training_version) >= Version("1.13"):
return "py39"

The default return value of the function should still be set to "py3".

It isn't. the difference is smoother logic processing with the same results as before, with the addition of py39 for 1.13 and above. the other logic was in reverse where it looked at lower versions before newer versions. this process has less logic hits as newer versions are used more than older. Anything under 1.5.0 still has request.param, between 1.9 and 1.5.0 uses py3 above 1.9 uses py38, same as old. the inclusion is 1.13 which is py39. this is valid and tested. Also, this is training and the code is also training's code which is used with inference as well.
Did notice it should be 1.5.0 and not just 1.5.. fixing that.

yeah I agree that this is a refactoring and doesn't change the code meaning. And sorry about the wrong lower version. The Version("1.5") and Version("1.5.0") have the same effect though.

kenny-ezirim · 2023-01-25T22:51:33Z

tests/conftest.py



 @pytest.fixture(scope="module", params=["py2", "py3"])
 def pytorch_inference_py_version(pytorch_inference_version, request):
-    if Version(pytorch_inference_version) < Version("1.4.0"):
-        return request.param
+    if Version(pytorch_inference_version) >= Version("1.13"):


Same applies to this function pytorch_inference_py_version as above:

should be 1.4.0 for inference. 1.5.0 for training. otherwise the new logic is fine.

thanks for catching this!

sagemaker-bot · 2023-01-25T23:00:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dcef8d4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-25T23:01:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: dcef8d4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-25T23:07:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dcef8d4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/image_uri_config/pytorch.json

sagemaker-bot · 2023-01-26T00:20:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:23:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:29:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 64bb0e8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:35:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:46:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 64bb0e8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:55:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:55:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:55:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T00:55:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-26T01:35:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 64bb0e8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Shibo Xing <[email protected]>

add PT 1.13.1 Inference

f34555b

ghost self-requested a review as a code owner January 12, 2023 17:33

ghost requested review from knikure and removed request for a team January 12, 2023 17:33

hballuru approved these changes Jan 12, 2023

View reviewed changes

knikure self-assigned this Jan 12, 2023

Shibo Xing and others added 2 commits January 23, 2023 22:54

chore: add PT-1.13.1 for tr

d4cdf64

Merge pull request #2 from ShiboXing/add-pt1.13.1-training

67eb6b2

chore: add PT-1.13.1 for tr

Merge branch 'master' into add-pt1.13.1-inference

bf203e3

Shibo Xing and others added 2 commits January 24, 2023 16:47

test: update pytorch training py_version configuration

238f1b8

solve py38 issue and training addition

9b25f3f

refactor get py version

1a6ad3c

knikure reviewed Jan 25, 2023

View reviewed changes

kenny-ezirim reviewed Jan 25, 2023

View reviewed changes

correct PT FW version for python logic

b0cbaf0

ShiboXing suggested changes Jan 25, 2023

View reviewed changes

src/sagemaker/image_uri_config/pytorch.json Outdated Show resolved Hide resolved

remove py2 from PT 1.4.0 Inference. accidental add

64bb0e8

kenny-ezirim approved these changes Jan 25, 2023

View reviewed changes

ShiboXing approved these changes Jan 26, 2023

View reviewed changes

knikure approved these changes Jan 26, 2023

View reviewed changes

knikure changed the title ~~Add PT 1.13.1 to SDK~~ feature: Add PyTorch 1.13.1 to SDK Jan 26, 2023

knikure merged commit 25c49d4 into aws:master Jan 26, 2023

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

feature: Add PyTorch 1.13.1 to SDK (aws#3587)

7c58598

Co-authored-by: Shibo Xing <[email protected]>

ghost deleted the add-pt1.13.1-inference branch March 31, 2023 20:25

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

feature: Add PyTorch 1.13.1 to SDK (aws#3587)

3db26b7

Co-authored-by: Shibo Xing <[email protected]>

feature: Add PyTorch 1.13.1 to SDK #3587

feature: Add PyTorch 1.13.1 to SDK #3587

Conversation

ghost commented Jan 12, 2023 • edited by ghost Loading

Merge Checklist

General

Tests

sagemaker-bot commented Jan 12, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 12, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 12, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 12, 2023

AWS CodeBuild CI Report

codecov-commenter commented Jan 12, 2023 • edited Loading

Codecov Report

sagemaker-bot commented Jan 12, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 23, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 23, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 23, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 23, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

knikure left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

kenny-ezirim Jan 25, 2023

Choose a reason for hiding this comment

ghost Jan 25, 2023 • edited by ghost Loading

Choose a reason for hiding this comment

ShiboXing Jan 25, 2023

Choose a reason for hiding this comment

kenny-ezirim Jan 25, 2023

Choose a reason for hiding this comment

ghost Jan 25, 2023

Choose a reason for hiding this comment

ShiboXing Jan 25, 2023

Choose a reason for hiding this comment

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 25, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jan 26, 2023

ghost commented Jan 12, 2023 •

edited by ghost

Loading

codecov-commenter commented Jan 12, 2023 •

edited

Loading

ghost Jan 25, 2023 •

edited by ghost

Loading