feature: add serverless inference image_uri support #3035

bhaoz · 2022-03-29T06:14:44Z

Issue #, if available:

Description of changes:
Image_uri.retrieve requires instance type now but that's not needed in serverless inference. To support this function, we default serverless inference to CPU instance type to get Amazon SageMaker first-party algorithm containers.

Testing done:
Add more integration tests

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter · 2022-03-29T06:31:57Z

Codecov Report

Merging #3035 (4282d85) into master (6ce11cb) will decrease coverage by 0.09%.
The diff coverage is 93.54%.

❗ Current head 4282d85 differs from pull request most recent head 05690bd. Consider uploading reports for the commit 05690bd to get more accurate results

@@            Coverage Diff             @@
##           master    #3035      +/-   ##
==========================================
- Coverage   89.47%   89.38%   -0.10%     
==========================================
  Files        1028      196     -832     
  Lines       84153    16661   -67492     
==========================================
- Hits        75294    14892   -60402     
+ Misses       8859     1769    -7090

Impacted Files	Coverage Δ
src/sagemaker/image_uris.py	`97.44% <60.00%> (-1.00%)`	⬇️
src/sagemaker/chainer/model.py	`97.56% <100.00%> (ø)`
src/sagemaker/huggingface/model.py	`79.10% <100.00%> (ø)`
src/sagemaker/model.py	`92.13% <100.00%> (ø)`
src/sagemaker/mxnet/model.py	`96.36% <100.00%> (ø)`
src/sagemaker/pytorch/model.py	`85.45% <100.00%> (ø)`
src/sagemaker/sklearn/model.py	`81.63% <100.00%> (ø)`
src/sagemaker/tensorflow/model.py	`90.32% <100.00%> (+0.10%)`	⬆️
src/sagemaker/xgboost/model.py	`97.43% <100.00%> (ø)`
src/sagemaker/session.py	`70.44% <0.00%> (-0.08%)`	⬇️
... and 832 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ce11cb...05690bd. Read the comment docs.

sagemaker-bot · 2022-03-29T06:41:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8434b4f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-29T07:32:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8434b4f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-29T07:32:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 8434b4f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-29T17:56:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3a9e118
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-29T18:57:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3a9e118
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-30T19:05:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 593d70a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-30T19:56:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 593d70a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-30T22:32:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b760bc2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-30T23:19:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b760bc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-30T23:45:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b760bc2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T00:04:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 05498db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T01:00:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 05498db
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T16:31:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d7ea758
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T16:55:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: bf6ecdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T17:37:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d7ea758
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T17:37:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: bf6ecdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T17:49:23Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: bf6ecdb
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-03-31T18:04:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d7ea758
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-01T17:59:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5813ab3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-01T19:10:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5813ab3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-01T20:15:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 805b366
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-01T21:22:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 805b366
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-04T17:18:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: bae2fbd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-04T17:52:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: bae2fbd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-04T19:47:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4282d85
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-04T20:24:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4282d85
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

do we need all this integ tests as we are adding support for image uris. will unittests not be enough.

navinsoni · 2022-04-05T19:33:14Z

doc/overview.rst

-For using SageMaker Serverless Inference, if you plan to use any of the SageMaker-provided container or Bring Your Own Container
-model, you will need to pass ``image_uri``. An example to use ``image_uri`` for creating MXNet model:
+For using SageMaker Serverless Inference, you can either use SageMaker-provided container or Bring Your Own Container model.
+An step by step example for using Serverless Inference with MXNet image :


nit: please change 'An' to 'A'

Resolved locally. Will wait other comments and update the PR

Changed the doc

doc/overview.rst

navinsoni · 2022-04-05T19:36:31Z

src/sagemaker/chainer/model.py

-                Chainer will be used. If ``framework_version`` or ``py_version``
+            image_uri (str): A Docker image URI (default: None). If not specified,
+                a default image for Chainer will be used.
+                If ``framework_version`` or ``py_version``
                are ``None``, then ``image_uri`` is required. If also ``None``,


please update what's none in "If also None"

This is the original docstring statement we have - API Version 2.78. Do you think we need to change this? Or may just keeping the original statement?

Let's change it as it will help later :)

bhaoz · 2022-04-05T21:17:56Z

Thanks Navin for reviewing this. But we need all those integ tests imo, they will ensure serverlss works well for all popular 1p algos. We didn't catch this image_uri inconvenience issue at the first place when we support serverless inference, and that could be avoid if we have comprehensive integ tests. (Since it only happens on some models)
Do you have any concerns for adding integ tests? Will that affect CI latency or other causing performance issue?

src/sagemaker/chainer/model.py

sagemaker-bot · 2022-04-06T22:40:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6baeb78
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni · 2022-04-06T23:01:21Z

src/sagemaker/chainer/model.py

-                Chainer will be used. If ``framework_version`` or ``py_version``
+            image_uri (str): A Docker image URI (default: None). If not specified,
+                a default image for Chainer will be used.
+                If ``framework_version`` or ``py_version``
                are ``None``, then ``image_uri`` is required. If also ``None``,


Let's change it as it will help later :)

sagemaker-bot · 2022-04-06T23:28:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7debf6a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-06T23:50:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6baeb78
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-07T00:12:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ad37e58
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-07T00:18:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7debf6a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-07T04:46:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7a5e501
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-07T05:28:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 05690bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-07T05:48:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7a5e501
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-04-07T06:02:06Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 05690bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

bhaoz changed the title ~~[WIP] feature: add serverless inference image_uri support~~ feature: add serverless inference image_uri support Mar 30, 2022

fm1ch4 approved these changes Mar 31, 2022

View reviewed changes

ahsan-z-khan requested review from mufaddal-rohawala and ahsan-z-khan April 4, 2022 19:23

navinsoni suggested changes Apr 5, 2022

View reviewed changes

mufaddal-rohawala reviewed Apr 5, 2022

View reviewed changes

src/sagemaker/chainer/model.py Show resolved Hide resolved

mufaddal-rohawala approved these changes Apr 6, 2022

View reviewed changes

ahsan-z-khan approved these changes Apr 6, 2022

View reviewed changes

navinsoni approved these changes Apr 6, 2022

View reviewed changes

feature: add serverless inference image_uri retrieve support

05690bd

ahsan-z-khan merged commit 338fd76 into aws:master Apr 7, 2022

ahsan-z-khan pushed a commit to ahsan-z-khan/sagemaker-python-sdk that referenced this pull request Apr 7, 2022

feature: add serverless inference image_uri retrieve support (aws#3035)

c86feb3

jerrypeng7773 pushed a commit to jerrypeng7773/sagemaker-python-sdk that referenced this pull request May 13, 2022

feature: add serverless inference image_uri retrieve support (aws#3035)

7a98d73

feature: add serverless inference image_uri support #3035

feature: add serverless inference image_uri support #3035

Conversation

bhaoz commented Mar 29, 2022 • edited Loading

Merge Checklist

General

Tests

codecov-commenter commented Mar 29, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Mar 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 29, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 30, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 30, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 30, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 30, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 30, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Mar 31, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 1, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Apr 4, 2022

AWS CodeBuild CI Report

navinsoni left a comment

Choose a reason for hiding this comment

navinsoni Apr 5, 2022

Choose a reason for hiding this comment

bhaoz Apr 5, 2022

Choose a reason for hiding this comment

bhaoz Apr 6, 2022

Choose a reason for hiding this comment

navinsoni Apr 5, 2022

Choose a reason for hiding this comment

bhaoz Apr 5, 2022

Choose a reason for hiding this comment

navinsoni Apr 6, 2022

Choose a reason for hiding this comment

bhaoz commented Apr 5, 2022 • edited Loading

sagemaker-bot commented Apr 6, 2022

AWS CodeBuild CI Report

navinsoni Apr 6, 2022

Choose a reason for hiding this comment

bhaoz commented Mar 29, 2022 •

edited

Loading

codecov-commenter commented Mar 29, 2022 •

edited

Loading

bhaoz commented Apr 5, 2022 •

edited

Loading