feature: make estimator accept json file as modelparallel config #3265

yongyanrao · 2022-07-26T14:56:38Z

Issue #, if available:

Description of changes: When passing distribution dict to estimator API PyTorch, we can let smp_config be either a dict (current behavior) or a string representing a config file.

            distribution = {
                "smdistributed": {
                    "modelparallel": {
                        "enabled": True,
                        "parameters": smp_config,
                    },
                },
                ...
            }

The config file needs to be a json file. e.g., params.json

{"partitions": 1, "tensor_parallel_degree": 2, "microbatches": 1, "optimize": "speed", "pipeline": "interleaved", "ddp": 1, "auto_partition": false, "default_partition": 0}

Testing done: local unit tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-07-26T15:04:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a741609
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T15:25:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a741609
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-07-26T15:26:24Z

Codecov Report

Merging #3265 (46ee21e) into master (1fa2377) will decrease coverage by 0.01%.
The diff coverage is 75.00%.

❗ Current head 46ee21e differs from pull request most recent head 05604ec. Consider uploading reports for the commit 05604ec to get more accurate results

@@            Coverage Diff             @@
##           master    #3265      +/-   ##
==========================================
- Coverage   89.17%   89.15%   -0.02%     
==========================================
  Files         204      204              
  Lines       18979    18922      -57     
==========================================
- Hits        16924    16870      -54     
+ Misses       2055     2052       -3

Impacted Files	Coverage Δ
src/sagemaker/fw_utils.py	`90.15% <75.00%> (-0.98%)`	⬇️
src/sagemaker/training_compiler/config.py	`82.05% <0.00%> (-1.29%)`	⬇️
src/sagemaker/huggingface/estimator.py	`93.18% <0.00%> (-1.27%)`	⬇️
src/sagemaker/model.py	`92.87% <0.00%> (-0.25%)`	⬇️
src/sagemaker/estimator.py	`89.41% <0.00%> (-0.10%)`	⬇️
src/sagemaker/workflow/clarify_check_step.py	`96.87% <0.00%> (-0.04%)`	⬇️
src/sagemaker/clarify.py	`95.65% <0.00%> (-0.03%)`	⬇️
src/sagemaker/workflow/quality_check_step.py	`99.20% <0.00%> (-0.02%)`	⬇️
src/sagemaker/inputs.py	`100.00% <0.00%> (ø)`
src/sagemaker/network.py	`100.00% <0.00%> (ø)`
... and 27 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-07-26T15:31:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a741609
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T15:42:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a741609
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T16:09:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a741609
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T02:18:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c121085
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T02:49:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: c121085
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T02:59:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c121085
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T03:18:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c121085
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T03:22:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: c121085
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T05:24:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9d575ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T05:29:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 9d575ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T05:44:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9d575ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T06:03:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9d575ab
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T07:47:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 9d575ab
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T21:43:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a42bfbd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T22:17:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a42bfbd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T22:26:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a42bfbd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T22:31:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a42bfbd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T22:41:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a42bfbd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/fw_utils.py

sagemaker-bot · 2022-08-04T17:21:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 046ee19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-04T17:27:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 046ee19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-04T17:44:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 046ee19
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-18T08:05:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 46ee21e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-18T09:30:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 46ee21e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-18T09:47:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 46ee21e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T17:18:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 748531d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T17:21:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 748531d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T17:45:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 748531d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T18:07:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 748531d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-10-19T21:35:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 748531d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

/bot run all

sagemaker-bot · 2022-11-16T19:18:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 05604ec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-16T19:23:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 05604ec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-16T19:30:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 05604ec
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-16T19:47:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 05604ec
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-16T19:49:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 05604ec
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…nfig

sagemaker-bot · 2022-12-01T22:40:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e67f733
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T22:45:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e67f733
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T22:57:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e67f733
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T23:53:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e67f733
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-02T00:56:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e67f733
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…#3265) Co-authored-by: Yongyan Rao <[email protected]>

yongyanrao force-pushed the file_smp_config_rebase branch from a741609 to c121085 Compare August 2, 2022 02:13

yongyanrao changed the title ~~feature: make estimator accept json and yaml file as modelparallel config~~ feature: make estimator accept json file as modelparallel config Aug 2, 2022

navinsoni force-pushed the file_smp_config_rebase branch from 9d575ab to a42bfbd Compare August 2, 2022 21:37

YangFei1990 approved these changes Aug 2, 2022

View reviewed changes

navinsoni force-pushed the file_smp_config_rebase branch from a42bfbd to 046ee19 Compare August 4, 2022 16:51

navinsoni approved these changes Aug 4, 2022

View reviewed changes

src/sagemaker/fw_utils.py Show resolved Hide resolved

claytonparnell force-pushed the file_smp_config_rebase branch from 46ee21e to 748531d Compare October 19, 2022 17:15

navinsoni force-pushed the file_smp_config_rebase branch from 748531d to 05604ec Compare November 16, 2022 18:36

navinsoni reviewed Nov 16, 2022

View reviewed changes

navinsoni self-assigned this Nov 16, 2022

Yongyan Rao added 2 commits December 1, 2022 14:04

feature: make estimator accept json and yaml file as modelparallel co…

1f20fbf

…nfig

Remove yaml for config file

e67f733

navinsoni force-pushed the file_smp_config_rebase branch from 05604ec to e67f733 Compare December 1, 2022 22:04

navinsoni merged commit d538219 into aws:master Dec 2, 2022

mizanfiu pushed a commit to mizanfiu/sagemaker-python-sdk that referenced this pull request Dec 13, 2022

feature: make estimator accept json file as modelparallel config (aws…

70c7050

…#3265) Co-authored-by: Yongyan Rao <[email protected]>

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

feature: make estimator accept json file as modelparallel config (aws…

8735745

…#3265) Co-authored-by: Yongyan Rao <[email protected]>

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

feature: make estimator accept json file as modelparallel config (aws…

a35ef92

…#3265) Co-authored-by: Yongyan Rao <[email protected]>

nmadan pushed a commit to nmadan/sagemaker-python-sdk that referenced this pull request Apr 18, 2023

feature: make estimator accept json file as modelparallel config (aws…

6cb8b30

…#3265) Co-authored-by: Yongyan Rao <[email protected]>

feature: make estimator accept json file as modelparallel config #3265

feature: make estimator accept json file as modelparallel config #3265

Conversation

yongyanrao commented Jul 26, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

codecov-commenter commented Jul 26, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2022

AWS CodeBuild CI Report

navinsoni left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Nov 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Nov 16, 2022

yongyanrao commented Jul 26, 2022 •

edited

Loading

codecov-commenter commented Jul 26, 2022 •

edited

Loading