Skip to content

Missing Tensorflow 2.9 inference image #3250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
plumdog opened this issue Jul 19, 2022 · 2 comments
Closed

Missing Tensorflow 2.9 inference image #3250

plumdog opened this issue Jul 19, 2022 · 2 comments

Comments

@plumdog
Copy link

plumdog commented Jul 19, 2022

Describe the bug

Tensorflow 2.9 inference image is reported as not available by this library, but does exist in registries.

To reproduce

>>> import sagemaker
>>> sagemaker.__version__
'2.99.0'
>>> sagemaker.image_uris.retrieve('tensorflow', 'eu-west-1', image_scope='inference', version='2.8', instance_type='ml.m5.large')
'763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu'
>>> sagemaker.image_uris.retrieve('tensorflow', 'eu-west-1', image_scope='inference', version='2.9', instance_type='ml.m5.large')
Traceback (most recent call last):
...
    raise ValueError(
ValueError: Unsupported tensorflow version: 2.9. You may need to upgrade your SDK version (pip install -U sagemaker) for newer tensorflow versions. Supported tensorflow version(s): 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.15.2, 1.15.3, 1.15.4, 1.15.5, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2, 2.4.1, 2.4.3, 2.5.1, 2.6.0, 2.6.3, 2.7.0, 2.8.0, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8.

But then, pulling images directly with Docker:

$ aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin "763104351884.dkr.ecr.eu-west-1.amazonaws.com"
Login Succeeded
$ docker pull 763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8.0-cpu                                                   
2.8.0-cpu: Pulling from tensorflow-inference
d5fd17ec1767: Downloading [=========>                                         ]  5.316MB/28.57MB
055d1b7bd083: Downloading [=>                                                 ]  4.855MB/214MB
501cf980eba1: Downloading [===>                                               ]  4.855MB/65.85MB
6eee2e438f6b: Pulling fs layer 
9b8cedc52042: Waiting 
c973b6111e73: Pulling fs layer 
8eda15d07ac8: Waiting 
d24fcb50554b: Waiting 
48896ee9d13a: Waiting 
c1c14bb3e866: Waiting 
29ea66b2c7c4: Pulling fs layer 
f27ea8b7dd4d: Pulling fs layer 
08b10d0bd12c: Waiting 
4fccd6276502: Waiting 
2a86a3199c71: Waiting 
37d47cae2d7e: Pulling fs layer 
af78b1cccabf: Waiting 
dcae54aa69ab: Waiting 
^C
$ docker pull 763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.9.0-cpu
2.9.0-cpu: Pulling from tensorflow-inference
d7bfe07ed847: Already exists 
ef6b4bb7a4ac: Downloading [=>                                                 ]  8.084MB/208.5MB
e365b2d13a81: Downloading [====>                                              ]  5.934MB/65.85MB
dc8a8d8bbf88: Download complete 
0dc26fa0c010: Downloading [================>                                  ]  10.03MB/30.2MB
63525e769512: Waiting 
5f7f441d968a: Waiting 
45742bfa386b: Waiting 
f20501310d53: Waiting 
7d3ab0cf075a: Waiting 
9bb4f6e81a92: Waiting 
b36eab1c1cbb: Waiting 
11ebb67b1885: Waiting 
75fc33f4c92d: Waiting 
31f4f6c8ee90: Waiting 
07a43246c874: Waiting 
57a569513401: Waiting 
572596e2e57a: Waiting

So the images exist in the registries and I am able to pull them, but this library doesn't know about them. This tracks with the current state of https://github.com/aws/sagemaker-python-sdk/blob/b4f05b86a90f4ae202ad7f9b048922ab490731fe/src/sagemaker/image_uri_config/tensorflow.json, which lists 2.9 as a training image, but not as an inference image.

Expected behavior

This library to know about the Tensorflow 2.9 inference image and for sagemaker.image_uris.retrieve('tensorflow', 'eu-west-1', image_scope='inference', version='2.9', instance_type='ml.m5.large') to return '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.9-cpu'

Screenshots or logs
See replication steps above.

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.99.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): Tensorflow
  • Framework version: 2.9
  • Python version: n/a
  • CPU or GPU: n/a
  • Custom Docker image (Y/N): N

Additional context
Add any other context about the problem here.

@arjkesh
Copy link
Contributor

arjkesh commented Nov 11, 2022

Hi @plumdog - thank you for reporting this issue. The sagemaker python SDK is unfortunately not updated with the TF2.9 images, though a PR is open to remedy the issue. We hope to get this in as soon as possible. In the meantime, if you are blocked on deploying a TF 2.9 model, you can use the "image_uri" parameter in the sagemaker model class to deploy TF2.9 models, and you can reference the same image URI you posted in this issue.

763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.9-cpu

Please let us know if there is any further info we can provide you in the short term - we will update this issue immediately after merging the TF2.9 image URI config changes.

@martinRenou
Copy link
Collaborator

Closing as fixed by #3465

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants