feat: jumpstart model id suggestions #2899

evakravi · 2022-02-04T20:15:49Z

Issue #, if available:

Description of changes:
This PR improves the error message when a bad model id is inputted for JumpStart utilities. If the model id does not exist for any versions, the error message indicates a closest match model id which can be used.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter · 2022-02-04T20:29:36Z

Codecov Report

Merging #2899 (b0a761b) into dev (21ac6fd) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##              dev    #2899   +/-   ##
=======================================
  Coverage   89.47%   89.48%           
=======================================
  Files         196      196           
  Lines       16584    16592    +8     
=======================================
+ Hits        14839    14847    +8     
  Misses       1745     1745

Impacted Files	Coverage Δ
src/sagemaker/jumpstart/validators.py	`91.86% <ø> (ø)`
src/sagemaker/jumpstart/cache.py	`96.58% <100.00%> (+0.25%)`	⬆️
src/sagemaker/jumpstart/types.py	`94.33% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21ac6fd...b0a761b. Read the comment docs.

sagemaker-bot · 2022-02-04T20:37:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 049a57c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-04T21:31:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 049a57c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-04T21:50:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 049a57c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

JGuinegagne

Minor comments/doc changes.

Also a question on how many model IDs suggestion do we surface.

JGuinegagne · 2022-02-07T20:51:15Z

src/sagemaker/jumpstart/cache.py

+        )
+        if other_model_id_version is not None:
+            error_msg += (
+                f"Consider using model id {model_id} with version "


"Consider using model ID '{model_id}' with version '{other_model_id_version}'."

JGuinegagne · 2022-02-07T20:52:38Z

src/sagemaker/jumpstart/cache.py

+
+        else:
+            possible_model_ids = [header.model_id for header in manifest.values()]
+            closest_model_id = get_close_matches(model_id, possible_model_ids, n=1, cutoff=0)[0]


question: should you show only one? or say for example: at least 1, up to 3 when score > xx ?

Let's just do 1 and make it simple. If we want to make this really good, we're better off just having a separate utility for searching model ids.

another question, can you get an IndexError here if get_close_matches returns an empty list?

If that's possible, could you handle this edge case and add a unit test for it?

No, because the cutoff is 0, there will always be a match.

Thanks. I know this is super edge case, but could there be a case where possible_model_ids is an empty list?

Only if the manifest is empty

JGuinegagne · 2022-02-07T21:03:24Z

src/sagemaker/jumpstart/cache.py

+        error_msg = f"Unable to find model manifest for {model_id} with version {version}. "
+
+        other_model_id_version = self._select_version(
+            "*", versions_incompatible_with_sagemaker


can you please add a comment as to why you use this variable.

reiterate: please explain why we use a variable called versions_incompatitble_with_sagemaker here.

JGuinegagne · 2022-02-07T21:07:15Z

src/sagemaker/jumpstart/cache.py

@@ -13,6 +13,7 @@
 """This module defines the JumpStartModelsCache class."""
 from __future__ import absolute_import
 import datetime
+from difflib import get_close_matches


sagemaker-bot · 2022-02-07T21:41:55Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4aa1670
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T22:40:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4aa1670
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-07T22:46:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4aa1670
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-08T17:46:21Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d66432e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-08T18:39:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d66432e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-08T18:52:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d66432e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-09T13:54:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c51d9bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-09T14:48:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c51d9bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-09T14:49:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c51d9bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-15T15:24:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2e017c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-15T16:18:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2e017c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-15T16:53:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2e017c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T14:01:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dded640
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T14:01:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5150498
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T14:34:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dded640
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T14:34:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5150498
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T14:45:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5150498
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T14:45:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dded640
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T17:33:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dded640
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T21:03:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5370f76
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-16T21:40:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5370f76
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-17T00:39:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5370f76
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-18T08:15:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5370f76
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-18T11:14:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5370f76
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-18T15:10:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b0a761b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-18T16:01:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b0a761b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-18T17:13:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b0a761b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-02-18T21:35:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b0a761b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Navin Soni <[email protected]> Co-authored-by: Mufaddal Rohawala <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: HappyAmazonian <[email protected]>

mufaddal-rohawala force-pushed the dev branch 2 times, most recently from 96c01d7 to c437191 Compare February 5, 2022 00:08

JGuinegagne requested changes Feb 7, 2022

View reviewed changes

evakravi force-pushed the feat/jumpstart-model-id-suggestions branch from 049a57c to 4aa1670 Compare February 7, 2022 21:25

jeniyat force-pushed the dev branch from 9d612a7 to bb7563f Compare February 7, 2022 23:40

evakravi force-pushed the feat/jumpstart-model-id-suggestions branch from 4aa1670 to d66432e Compare February 8, 2022 17:21

jeniyat force-pushed the dev branch from 7ea4b80 to bb7563f Compare February 9, 2022 00:11

evakravi added 2 commits February 9, 2022 13:33

feat: jumpstart model id suggestions

598562b

chore: resolve git comments

c51d9bd

evakravi force-pushed the feat/jumpstart-model-id-suggestions branch from d66432e to c51d9bd Compare February 9, 2022 13:34

navinsoni and others added 2 commits February 15, 2022 00:30

fix: Update Static Endpoint (aws#2931)

b591959

fix: hyperparameter validation error messages

2e017c4

navinsoni and others added 2 commits February 15, 2022 18:27

Add exception in test_action (aws#2938)

3c5ea3a

change: pin test dependencies (aws#2929)

9d84e2e

qidewenwhen and others added 4 commits February 15, 2022 21:30

feature: Add FailStep Support for Sagemaker Pipeline (aws#2872)

cede5fa

change: use recommended inference image uri from Neo API (aws#2923)

a765512

change: add url to model id table for jumpstart

5150498

Merge branch 'dev' into feat/jumpstart-model-id-suggestions

dded640

JGuinegagne approved these changes Feb 16, 2022

View reviewed changes

navinsoni force-pushed the dev branch from 58cf626 to 46391bf Compare February 16, 2022 19:55

Merge branch 'dev' into feat/jumpstart-model-id-suggestions

5370f76

shreyapandit approved these changes Feb 17, 2022

View reviewed changes

navinsoni approved these changes Feb 18, 2022

View reviewed changes

Merge branch 'dev' into feat/jumpstart-model-id-suggestions

b0a761b

navinsoni merged commit 09d3fd1 into aws:dev Feb 18, 2022

feat: jumpstart model id suggestions #2899

feat: jumpstart model id suggestions #2899

Conversation

evakravi commented Feb 4, 2022 • edited Loading

Merge Checklist

General

Tests

codecov-commenter commented Feb 4, 2022 • edited Loading

Codecov Report

sagemaker-bot commented Feb 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 4, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 4, 2022

AWS CodeBuild CI Report

JGuinegagne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sagemaker-bot commented Feb 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 7, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 8, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 9, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 9, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 9, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 15, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 16, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 17, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 18, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Feb 18, 2022

evakravi commented Feb 4, 2022 •

edited

Loading

codecov-commenter commented Feb 4, 2022 •

edited

Loading