fix: Add gpu capability to local #3136

teticio · 2022-05-25T19:47:54Z

Issue #, if available:

Description of changes:

Currently local_gpu gives an error of Unknown runtime specified nvidia when running docker-compose. This no longer works with docker-compose version 1.27.0+. Now the way to enable GPUs according to the docker-compose documentation is to add the following config to the docker-compose.yml file:

services:
    ...
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

Testing done:

Ran tox tests/unit

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-05-25T19:51:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6bf4402
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-25T20:24:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6bf4402
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-25T20:38:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6bf4402
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-25T20:56:29Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6bf4402
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-25T22:40:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6bf4402
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T08:23:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2c37ea2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T08:54:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2c37ea2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T09:07:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2c37ea2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T09:21:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2c37ea2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T09:36:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2c37ea2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T10:35:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6c0f7fd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-05-31T10:35:26Z

Codecov Report

Merging #3136 (f402c9d) into master (6d7bfc4) will decrease coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3136      +/-   ##
==========================================
- Coverage   89.69%   89.62%   -0.07%     
==========================================
  Files        1048      200     -848     
  Lines       87791    17387   -70404     
==========================================
- Hits        78742    15583   -63159     
+ Misses       9049     1804    -7245

Impacted Files	Coverage Δ
src/sagemaker/local/image.py	`75.89% <100.00%> (ø)`
...lib/python3.6/site-packages/sagemaker/vpc_utils.py
...thon3.6/site-packages/sagemaker/apiutils/_utils.py
...r/cli/compatibility/v2/modifiers/tf_legacy_mode.py
...lib/python3.7/site-packages/sagemaker/predictor.py
...ages/sagemaker/feature_store/feature_definition.py
...gemaker/model_monitor/cron_expression_generator.py
...ib/python3.6/site-packages/sagemaker/amazon/knn.py
...y36/lib/python3.6/site-packages/sagemaker/utils.py
...hon3.6/site-packages/sagemaker/chainer/__init__.py
... and 840 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d7bfc4...f402c9d. Read the comment docs.

sagemaker-bot · 2022-05-31T10:47:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 6c0f7fd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T10:58:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6c0f7fd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T11:16:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6c0f7fd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T11:21:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6c0f7fd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T15:28:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f402c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T15:41:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f402c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T15:53:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f402c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T16:11:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f402c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-05-31T16:19:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f402c9d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2022-05-31T17:12:05Z

@teticio, thanks so much for contributing to fixing this, PR changes looks good to me. Were you able to verify this to work locally for gpu?

teticio · 2022-05-31T18:54:59Z

Yes, I was able to run inference on my GPU locally. Thanks for reviewing this.

…

On Tue, 31 May 2022 at 18:12, Mufaddal Rohawala ***@***.***> wrote: @teticio <https://github.com/teticio>, thanks so much for contributing to fixing this, PR changes looks good to me. Were you able to verify this to work locally for gpu? — Reply to this email directly, view it on GitHub <#3136 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKRPDBY3OAQQPHDRDNZF4LDVMZB67ANCNFSM5W6IJTHQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

teticio · 2022-06-21T17:13:41Z

I expect this is pretty low priority but just wondering if it's likely to get reviewed any time soon or if I need to do anything else right now. Thanks.

teticio · 2022-08-03T08:10:04Z

Sorry to bug you @mufaddal-rohawala, but I was wondering if this might get reviewed sometime? Thanks

navinsoni · 2022-08-04T17:53:14Z

@teticio Thank you for submitting the change.

teticio added 2 commits May 25, 2022 17:27

feature: add GPU capability to local mode

5b887e7

bug: local_gpu gives error Unknown runtime specified nvidia

6bf4402

Merge branch 'aws:master' into add-gpu-capability-to-local

2c37ea2

fix line too long in local_gpu case

6c0f7fd

ammended test for local_gpu

f402c9d

navinsoni approved these changes Aug 4, 2022

View reviewed changes

navinsoni changed the title ~~Add gpu capability to local~~ fix: Add gpu capability to local Aug 4, 2022

navinsoni merged commit 9649420 into aws:master Aug 4, 2022

JoseJuan98 pushed a commit to JoseJuan98/sagemaker-python-sdk that referenced this pull request Mar 4, 2023

fix: Add gpu capability to local (aws#3136)

cba4c20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Add gpu capability to local #3136

fix: Add gpu capability to local #3136

teticio commented May 25, 2022 •

edited

Loading

sagemaker-bot commented May 25, 2022

sagemaker-bot commented May 25, 2022

sagemaker-bot commented May 25, 2022

sagemaker-bot commented May 25, 2022

sagemaker-bot commented May 25, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

codecov-commenter commented May 31, 2022 •

edited

Loading

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

sagemaker-bot commented May 31, 2022

mufaddal-rohawala commented May 31, 2022

teticio commented May 31, 2022 via email

teticio commented Jun 21, 2022

teticio commented Aug 3, 2022

navinsoni commented Aug 4, 2022

fix: Add gpu capability to local #3136

fix: Add gpu capability to local #3136

Conversation

teticio commented May 25, 2022 • edited Loading

Merge Checklist

General

Tests

sagemaker-bot commented May 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 25, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

codecov-commenter commented May 31, 2022 • edited Loading

Codecov Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented May 31, 2022

AWS CodeBuild CI Report

mufaddal-rohawala commented May 31, 2022

teticio commented May 31, 2022 via email

teticio commented Jun 21, 2022

teticio commented Aug 3, 2022

navinsoni commented Aug 4, 2022

teticio commented May 25, 2022 •

edited

Loading

codecov-commenter commented May 31, 2022 •

edited

Loading