Feature: support torchrun for gpu instances #3672

yl-to · 2023-02-21T20:12:09Z

Issue #, if available:
Expanding Torchrun support from Trainum instances to all GPU instances.
Description of changes:

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-02-21T20:16:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c56d79d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T20:49:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c56d79d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T21:00:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c56d79d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T21:08:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c56d79d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T21:19:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: c56d79d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run unit-tests

sagemaker-bot · 2023-02-21T21:57:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c56d79d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T23:08:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: cce46ad
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T23:37:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: cce46ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T23:45:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: cce46ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T23:48:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: cce46ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-21T23:58:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: cce46ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-02-22T01:14:17Z

Codecov Report

Merging #3672 (77a89ed) into master (48a90a5) will decrease coverage by 0.78%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3672      +/-   ##
==========================================
- Coverage   89.73%   88.96%   -0.78%     
==========================================
  Files         968      227     -741     
  Lines       91056    22537   -68519     
==========================================
- Hits        81712    20049   -61663     
+ Misses       9344     2488    -6856

Impacted Files	Coverage Δ
src/sagemaker/fw_utils.py	`90.96% <100.00%> (ø)`
...y39/lib/python3.9/site-packages/sagemaker/tuner.py
....9/site-packages/sagemaker/tensorflow/estimator.py
...ckages/sagemaker/model_monitor/model_monitoring.py
...3.7/site-packages/sagemaker/workflow/parameters.py
...hon3.9/site-packages/sagemaker/pytorch/defaults.py
...ython3.10/site-packages/sagemaker/deserializers.py
...ckages/sagemaker/inference_recommender/__init__.py
...thon3.7/site-packages/sagemaker/hyperparameters.py
...b/python3.8/site-packages/sagemaker/rl/__init__.py
... and 1186 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-02-22T01:49:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 97d919c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-22T01:55:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 97d919c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-22T01:59:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 97d919c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-22T02:09:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 97d919c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-22T02:20:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 97d919c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/fw_utils.py

sagemaker-bot · 2023-02-28T01:41:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ea567d4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T03:38:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ea567d4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…into torchrun_gpu_sup

sagemaker-bot · 2023-02-28T10:12:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4b0e14c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T10:54:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4b0e14c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T10:55:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4b0e14c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T11:04:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4b0e14c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T11:29:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4b0e14c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run local-mode-tests, pr, slow-tests

sagemaker-bot · 2023-02-28T18:02:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4b0e14c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…into torchrun_gpu_sup

sagemaker-bot · 2023-02-28T18:22:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T18:57:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T19:04:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 77a89ed
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T19:15:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 77a89ed
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T19:21:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run local-mode-tests, pr, slow-tests

sagemaker-bot · 2023-02-28T19:29:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run local-mode-tests

sagemaker-bot · 2023-02-28T20:09:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T20:25:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-02-28T20:30:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

trajanikant

/bot run local-mode-tests, pr, slow-tests

sagemaker-bot · 2023-02-28T23:43:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-01T00:22:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-01T00:38:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 77a89ed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

feature: support torchrun for gpu instances

c56d79d

yl-to requested a review from a team as a code owner February 21, 2023 20:12

yl-to requested review from trajanikant and removed request for a team February 21, 2023 20:12

trajanikant reviewed Feb 21, 2023

View reviewed changes

fix: format fix

cce46ad

fix: logic fix

97d919c

josephevans reviewed Feb 22, 2023

View reviewed changes

src/sagemaker/fw_utils.py Show resolved Hide resolved

claytonparnell assigned trajanikant Feb 22, 2023

yl-to added 3 commits February 22, 2023 15:01

feature: support torchrun for gpu instances

334e64e

fix: format fix

40a13fe

fix: logic fix

a2e2493

trajanikant force-pushed the torchrun_gpu_sup branch from 97d919c to a2e2493 Compare February 22, 2023 23:01

Merge branch 'master' of https://github.com/yl-to/sagemaker-python-sdk …

4b0e14c

…into torchrun_gpu_sup

josephevans approved these changes Feb 28, 2023

View reviewed changes

trajanikant reviewed Feb 28, 2023

View reviewed changes

Merge branch 'master' of https://github.com/yl-to/sagemaker-python-sdk …

77a89ed

…into torchrun_gpu_sup

trajanikant reviewed Feb 28, 2023

View reviewed changes

trajanikant merged commit 888f45c into aws:master Mar 1, 2023

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

Feature: support torchrun for gpu instances (aws#3672)

d69a8bd

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

Feature: support torchrun for gpu instances (aws#3672)

1771d24

Feature: support torchrun for gpu instances #3672

Feature: support torchrun for gpu instances #3672

Conversation

yl-to commented Feb 21, 2023

Merge Checklist

General

Tests

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

trajanikant left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 21, 2023

AWS CodeBuild CI Report

codecov-commenter commented Feb 22, 2023 • edited Loading

Codecov Report

sagemaker-bot commented Feb 22, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 22, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 22, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 22, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 22, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

trajanikant left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

trajanikant left a comment • edited Loading

Choose a reason for hiding this comment

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

trajanikant left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Feb 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Feb 28, 2023

codecov-commenter commented Feb 22, 2023 •

edited

Loading

trajanikant left a comment •

edited

Loading