change: add a check to prevent launching a modelparallel job on CPU only instances #3262

yongyanrao · 2022-07-22T21:07:08Z

Issue #, if available:

*Description of changes: Added a check to prevent launching a modelparallel job on CPU only instances.

*Testing done:

All unit tests passed after the change (We ran unit tests in a conda environment with python=3.8. Please find the following results). Since we only add a check by directly using an existing API, we would like to discuss with the reviewers to see if a new unit test is necessary.

$ tox tests/unit
...
_________________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________________
  black-format: commands succeeded
  flake8: commands succeeded
  pylint: commands succeeded
  docstyle: commands succeeded
  sphinx: commands succeeded
  doc8: commands succeeded
  twine: commands succeeded
  py37: commands succeeded
  py38: commands succeeded
ERROR:  py39: InterpreterNotFound: python3.9
ERROR:  py310: InterpreterNotFound: python3.10

Live tests
-- Failure case

% python sagemaker_train_script.py --model pytorch-gpt2-precision --ddp 1 --tag pytorch_cur_testtp-preview-0-1 --partitions 2 --instance_type t2.nano --processes_per_host 8 --tensor_parallel_degree 2 --instance_count 2 --microbatches 2
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
AWS account:…
AWS region:…
Training image:  …
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
WARNING:sagemaker.deprecations:The class sagemaker.session.s3_input has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
WARNING:sagemaker.deprecations:The class sagemaker.session.s3_input has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
Creating trial pt-gpt2-pytorch-cur-testtp-preview-0-1-t2-nano-2-seq-1-ddp-cluster-tp2--shard0--partial0-2022-07-19-01-48-40 under experiment rubik-pipeline
checkpoint s3 bucket: s3://rubik-mp-benchmarks/output/pytorch_gpt2/logits_shard_0_pytorch-cur-testtp-preview-0-1-t2-nano-2-seq-1-ddp/
Traceback (most recent call last):
  File "sagemaker_train_script.py", line 1982, in <module>
    main()
  File "sagemaker_train_script.py", line 1966, in main
    pytorch_gpt2_precision(args.tensor_parallel_degree, args.shard_optimizer_state)
  File "sagemaker_train_script.py", line 740, in pytorch_gpt2_precision
    pytorch_estimator = PyTorch(
  File "/Users/…/miniconda3/envs/test_sagemaker/lib/python3.8/site-packages/sagemaker/pytorch/estimator.py", line 211, in __init__
    validate_distribution_instance(self.sagemaker_session, distribution, instance_type)
  File "/Users/…/miniconda3/envs/test_sagemaker/lib/python3.8/site-packages/sagemaker/fw_utils.py", line 732, in validate_distribution_instance
    raise ValueError(
ValueError: modelparallel only runs on GPU-enabled instances. t2.nano does not support GPU.

-- Success case

% python sagemaker_train_script.py --model pytorch-gpt2-precision --ddp 1 --tag pytorch_cur_testtp-preview-0-1 --partitions 2 --instance_type p4d.24xlarge --processes_per_host 8 --tensor_parallel_degree 2 --instance_count 2 --microbatches 2
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
AWS account:...
AWS region:...
Training image:  ...
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
WARNING:sagemaker.deprecations:The class sagemaker.session.s3_input has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
WARNING:sagemaker.deprecations:The class sagemaker.session.s3_input has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
Creating trial pt-gpt2-pytorch-cur-testtp-preview-0-1-p4d-24xlarge-2-seq-1-ddp-cluster-tp2--shard0--partial0-2022-07-19-02-18-56 under experiment rubik-pipeline
checkpoint s3 bucket: s3://rubik-mp-benchmarks/output/pytorch_gpt2/logits_shard_0_pytorch-cur-testtp-preview-0-1-p4d-24xlarge-2-seq-1-ddp/
INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: latest.
INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.
INFO:sagemaker:Creating training-job with name: rubik-pytorch-cur-testtp-preview-0-1-p4d-24xlar-1658197137-2b71
2022-07-19 02:18:59 Starting - Starting the training job..

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-07-25T21:56:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d6893c1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-25T22:28:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d6893c1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-25T22:37:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e834583
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-25T23:07:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: e834583
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-25T23:19:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e834583
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-25T23:21:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e834583
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-25T23:46:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: e834583
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T04:12:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5207ddc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-07-26T04:13:33Z

Codecov Report

Merging #3262 (d74ce08) into master (38b4916) will decrease coverage by 0.60%.
The diff coverage is 100.00%.

❗ Current head d74ce08 differs from pull request most recent head a2e28f9. Consider uploading reports for the commit a2e28f9 to get more accurate results

@@            Coverage Diff             @@
##           master    #3262      +/-   ##
==========================================
- Coverage   89.40%   88.80%   -0.61%     
==========================================
  Files         848      203     -645     
  Lines       72563    17992   -54571     
==========================================
- Hits        64877    15977   -48900     
+ Misses       7686     2015    -5671

Impacted Files	Coverage Δ
src/sagemaker/fw_utils.py	`90.13% <100.00%> (+0.46%)`	⬆️
src/sagemaker/pytorch/estimator.py	`100.00% <100.00%> (ø)`
...s/sagemaker/workflow/pipeline_experiment_config.py
...b/python3.9/site-packages/sagemaker/serializers.py
...hon3.7/site-packages/sagemaker/xgboost/defaults.py
...3.8/site-packages/sagemaker/serverless/__init__.py
...3.9/site-packages/sagemaker/workflow/conditions.py
...ython3.9/site-packages/sagemaker/spark/__init__.py
...-packages/sagemaker/workflow/quality_check_step.py
...hon3.8/site-packages/sagemaker/xgboost/__init__.py
... and 637 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

sagemaker-bot · 2022-07-26T04:19:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 5207ddc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T04:34:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5207ddc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T04:57:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5207ddc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T06:36:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5207ddc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-26T23:20:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5207ddc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T22:50:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: be9f546
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:30:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: be9f546
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:32:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: be9f546
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-28T23:38:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2c51639
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:10:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d74ce08
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:18:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d74ce08
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:32:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d74ce08
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T00:54:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d74ce08
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T02:34:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d74ce08
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T04:38:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d74ce08
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T05:40:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d74ce08
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T20:22:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d74ce08
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-07-29T20:23:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d74ce08
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T21:39:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a1e90df
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T22:00:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a1e90df
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T22:21:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a1e90df
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T22:22:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a1e90df
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T22:22:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a1e90df
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…nly instances.

sagemaker-bot · 2022-08-01T23:27:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a2e28f9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T23:46:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a2e28f9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-01T23:57:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a2e28f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T00:12:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a2e28f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T00:14:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a2e28f9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-08-02T01:49:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a2e28f9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…on CPU only instances (#3262)" This reverts commit 7d30d8c.

…nly instances (aws#3262) Co-authored-by: Yongyan Rao <[email protected]>

yongyanrao force-pushed the cpu_check branch from d6893c1 to e834583 Compare July 25, 2022 22:32

yongyanrao requested a review from qidewenwhen July 25, 2022 23:57

yongyanrao force-pushed the cpu_check branch from e834583 to 5207ddc Compare July 26, 2022 03:44

yongyanrao force-pushed the cpu_check branch from be9f546 to 2c51639 Compare July 28, 2022 23:33

yongyanrao force-pushed the cpu_check branch from 2c51639 to d74ce08 Compare July 28, 2022 23:42

mufaddal-rohawala approved these changes Jul 29, 2022

View reviewed changes

yongyanrao removed the request for review from qidewenwhen July 29, 2022 15:21

navinsoni force-pushed the cpu_check branch from d74ce08 to a1e90df Compare August 1, 2022 21:11

navinsoni force-pushed the cpu_check branch from a1e90df to 1029251 Compare August 1, 2022 22:37

Yongyan Rao added 2 commits August 1, 2022 17:57

change: add a check to prevent launching a modelparallel job on CPU o…

82e2a94

…nly instances.

Add unit tests

a2e28f9

yongyanrao force-pushed the cpu_check branch from 1029251 to a2e28f9 Compare August 1, 2022 22:59

navinsoni merged commit 7d30d8c into aws:master Aug 2, 2022

navinsoni added a commit that referenced this pull request Aug 4, 2022

Revert "change: add a check to prevent launching a modelparallel job …

01be840

…on CPU only instances (#3262)" This reverts commit 7d30d8c.

navinsoni mentioned this pull request Aug 4, 2022

fix: Revert "change: add a check to prevent launching a modelparallel job on CPU only instances" #3280

Merged

knikure pushed a commit to knikure/sagemaker-python-sdk that referenced this pull request Aug 4, 2022

change: add a check to prevent launching a modelparallel job on CPU o…

2c28949

…nly instances (aws#3262) Co-authored-by: Yongyan Rao <[email protected]>

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

change: add a check to prevent launching a modelparallel job on CPU o…

e368be0

…nly instances (aws#3262) Co-authored-by: Yongyan Rao <[email protected]>

change: add a check to prevent launching a modelparallel job on CPU only instances #3262

change: add a check to prevent launching a modelparallel job on CPU only instances #3262

Conversation

yongyanrao commented Jul 22, 2022

Merge Checklist

General

Tests

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

codecov-commenter commented Jul 26, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 26, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 28, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jul 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Aug 2, 2022

codecov-commenter commented Jul 26, 2022 •

edited

Loading