[change]: Support huggingface hub model_id for DJL Models #3753

siddvenk · 2023-03-24T18:37:31Z

Issue #, if available:
N/A

Description of changes:
This changes adds support for deploying models via DJL using the HuggingFace Hub model_id. Users can either supply a valid model_id, or an s3 url pointing to model artifacts in order to deploy DJLModels on SageMaker.

Testing done:
Endpoint testing done for gpt-j-6b, stable-diffusion-2-base, opt-2.7b using both hf hub model id, and s3 model artifacts (both engines as well).

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2023-03-24T18:42:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 914a656
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

lanking520 · 2023-03-24T19:19:35Z

src/sagemaker/djl_inference/model.py

+    my_env["GIT_TERMINAL_PROMPT"] = "0"
+    model_config = None
+    with TemporaryDirectory() as tmp_dir:
+        subprocess.check_call(["git", "clone", git_lfs_model_repo, tmp_dir], env=my_env)


will this clone the full model artifacts?

No, the GIT_LFS_SKIP_SMUDGE=1 will cause git to download pointers to the large files. This is still a bit overkill since it will download all the json/txt files.

I don't see any built in utils to just download a single file from git, but another option could be
curl https://huggingface.co/<repo/model>/raw/main/config.json > config.json

I chose the git option (though likely not the optimal choice) for 2 reasons.

This sdk already assumes that the user has git installed wherever they are running the code. curl or wget is more efficient to just get the single file, but i'm not sure whether we should assume user has that installed wherever they are running this from.

will the default branch for all model repos be main? This might be a fair assumption to make, I haven't seen other names for the primary branch yet.

Thoughts?

sagemaker-bot · 2023-03-24T19:30:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 914a656
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T19:30:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 914a656
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T19:35:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 914a656
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T20:03:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: ad9fd7a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T20:33:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: ad9fd7a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T20:44:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: ad9fd7a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T20:48:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: ad9fd7a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-24T20:52:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: ad9fd7a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-03-27T18:30:25Z

Codecov Report

Merging #3753 (ad9fd7a) into master (c02d74b) will decrease coverage by 0.77%.
The diff coverage is 100.00%.

❗ Current head ad9fd7a differs from pull request most recent head f8c83ab. Consider uploading reports for the commit f8c83ab to get more accurate results

@@            Coverage Diff             @@
##           master    #3753      +/-   ##
==========================================
- Coverage   89.84%   89.08%   -0.77%     
==========================================
  Files         984      231     -753     
  Lines       92360    22873   -69487     
==========================================
- Hits        82984    20377   -62607     
+ Misses       9376     2496    -6880

Impacted Files	Coverage Δ
src/sagemaker/djl_inference/defaults.py	`100.00% <100.00%> (ø)`
src/sagemaker/djl_inference/model.py	`93.11% <100.00%> (ø)`
src/sagemaker/workflow/emr_step.py	`100.00% <100.00%> (ø)`

... and 1212 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2023-03-27T18:33:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4a3df39
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-27T18:35:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4a3df39
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-27T19:46:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 310ac7f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-27T19:51:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 310ac7f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-27T19:53:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 310ac7f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-27T19:53:40Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 310ac7f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-27T21:50:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 310ac7f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

lanking520 · 2023-03-27T23:05:25Z

tests/unit/test_djl_inference.py

+    )
+    assert model.engine == DJLServingEngineEntryPointDefaults.DEEPSPEED
+    expected_git_url = f"https://huggingface.co/{HF_MODEL_ID}/raw/main/config.json"
+    subprocess_run.assert_any_call(["curl", expected_git_url], capture_output=True, check=True)


use urllib or urllib3

lanking520 · 2023-03-27T23:05:35Z

tests/unit/test_djl_inference.py

        )
-    assert str(invalid_s3_data.value).startswith("DJLModel only supports loading model artifacts")
+        subprocess_run.assert_any_call(["curl", expected_git_url], capture_output=True, check=True)


sagemaker-bot · 2023-03-28T00:55:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6ad4b69
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-28T00:58:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 6ad4b69
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-28T01:37:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f8c83ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-28T01:49:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f8c83ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-28T01:52:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f8c83ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-28T01:55:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f8c83ab
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-03-28T03:52:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f8c83ab
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

lanking520

LGTM!

siddvenk · 2023-03-30T16:47:24Z

@claytonparnell the slow tests are failing - is this still a known issue with the tests? Looking at the logs the errors are unrelated to the changes here.

If so, are we ok to merge this PR?

siddvenk · 2023-04-05T17:50:56Z

Hi @claytonparnell, can we merge this PR?

siddvenk requested a review from a team as a code owner March 24, 2023 18:37

siddvenk requested review from claytonparnell and removed request for a team March 24, 2023 18:37

lanking520 reviewed Mar 24, 2023

View reviewed changes

siddvenk force-pushed the djl-hf-model-id branch from 914a656 to ad9fd7a Compare March 24, 2023 19:56

siddvenk force-pushed the djl-hf-model-id branch from ad9fd7a to 4a3df39 Compare March 27, 2023 18:29

siddvenk force-pushed the djl-hf-model-id branch from 4a3df39 to 310ac7f Compare March 27, 2023 18:57

lanking520 reviewed Mar 27, 2023

View reviewed changes

siddvenk force-pushed the djl-hf-model-id branch from 310ac7f to 6ad4b69 Compare March 28, 2023 00:50

[change]: Support huggingface hub model_id for DJL Models

f8c83ab

siddvenk force-pushed the djl-hf-model-id branch from 6ad4b69 to f8c83ab Compare March 28, 2023 01:00

lanking520 approved these changes Mar 28, 2023

View reviewed changes

claytonparnell merged commit c4c4e83 into aws:master Apr 5, 2023

doddaspk-amzn pushed a commit to doddaspk-amzn/sagemaker-python-sdk that referenced this pull request Apr 6, 2023

feature: Support huggingface hub model_id for DJL Models (aws#3753)

50176b1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[change]: Support huggingface hub model_id for DJL Models #3753

[change]: Support huggingface hub model_id for DJL Models #3753

siddvenk commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

lanking520 Mar 24, 2023

siddvenk Mar 24, 2023 •

edited

Loading

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

sagemaker-bot commented Mar 24, 2023

codecov-commenter commented Mar 27, 2023 •

edited

Loading

sagemaker-bot commented Mar 27, 2023

sagemaker-bot commented Mar 27, 2023

sagemaker-bot commented Mar 27, 2023

sagemaker-bot commented Mar 27, 2023

sagemaker-bot commented Mar 27, 2023

sagemaker-bot commented Mar 27, 2023

sagemaker-bot commented Mar 27, 2023

lanking520 Mar 27, 2023

siddvenk Mar 28, 2023

lanking520 Mar 27, 2023

siddvenk Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

sagemaker-bot commented Mar 28, 2023

lanking520 left a comment

siddvenk commented Mar 30, 2023

siddvenk commented Apr 5, 2023

[change]: Support huggingface hub model_id for DJL Models #3753

[change]: Support huggingface hub model_id for DJL Models #3753

Conversation

siddvenk commented Mar 24, 2023

Merge Checklist

General

Tests

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

lanking520 Mar 24, 2023

Choose a reason for hiding this comment

siddvenk Mar 24, 2023 • edited Loading

Choose a reason for hiding this comment

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 24, 2023

AWS CodeBuild CI Report

codecov-commenter commented Mar 27, 2023 • edited Loading

Codecov Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 27, 2023

AWS CodeBuild CI Report

lanking520 Mar 27, 2023

Choose a reason for hiding this comment

siddvenk Mar 28, 2023

Choose a reason for hiding this comment

lanking520 Mar 27, 2023

Choose a reason for hiding this comment

siddvenk Mar 28, 2023

Choose a reason for hiding this comment

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Mar 28, 2023

AWS CodeBuild CI Report

lanking520 left a comment

Choose a reason for hiding this comment

siddvenk commented Mar 30, 2023

siddvenk commented Apr 5, 2023

siddvenk Mar 24, 2023 •

edited

Loading

codecov-commenter commented Mar 27, 2023 •

edited

Loading