feature: JumpStart CuratedHub class creation and function definitions #4448

jinyoung-lim · 2024-02-22T21:25:50Z

Issue #, if available:

Description of changes:

Set up class and function definitions for CuratedHub .

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
[] I have checked that my tests are not configured for a specific region or account (if appropriate)
[] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

Ran below locally:

black -l 100 .               
flake8
python -m pylint --rcfile=.pylintrc -j 0 src/**/sagemaker/**/jumpstart/*hub*
python -m pytest tests/unit/**/jumpstart -W ignore::DeprecationWarning

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

mufaddal-rohawala · 2024-02-22T21:31:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4922511
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

evakravi · 2024-02-22T21:35:21Z

src/sagemaker/jumpstart/curated_hub/utils.py

+    return f"sagemaker-hubs-{region}-{account_id}"
+
+
+def create_hub(


why not add this to sagemaker.session module? It seems more appropriate there

Please see PR description but after having a discussion with @bencrabtree , we want to keep the session HubAPI calls to be just a bare-bone wrapper for Hub API calls and have hubutils to handle any custom logics.

evakravi · 2024-02-22T21:37:07Z

src/sagemaker/jumpstart/curated_hub/utils.py

+        document_schema_version,
+        hub_name,
+        hub_content_name,
+        hub_content_type,
+        hub_content_document,
+        hub_content_display_name,
+        hub_content_description,
+        hub_content_version,
+        hub_content_markdown,
+        hub_content_search_keywords,
+        tags,


for this many arguments, can we use kwarg style args? hub_name=hub_name, ...

Good point. Will do.

bencrabtree · 2024-02-22T21:37:14Z

src/sagemaker/jumpstart/curated_hub/utils.py

+    """
+
+    if hub_bucket_name is None:
+        hub_bucket_name = _generate_default_hub_bucket_name(sagemaker_session)


We need to create the default bucket here too if we're going to have a default value. I'd actually suggest to make the value required until this is the case

Hmmm that is a good point. Will handle one way or the other.

bencrabtree · 2024-02-22T21:38:02Z

src/sagemaker/jumpstart/curated_hub/utils.py

+
+def list_hub_contents(
+    hub_name: str,
+    hub_content_type: HubDataType.MODEL or HubDataType.NOTEBOOK,


Can this type be just HubDataType?

Edit: we actually should rename it HubContentType

HubDataType also has HUB

mufaddal-rohawala · 2024-02-22T21:41:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4922511
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-22T22:31:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4922511
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-22T22:40:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 4922511
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-22T23:30:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 4922511
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-23T00:23:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4922511
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-23T01:01:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 45a4a4d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-23T01:04:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 45a4a4d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-23T01:04:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 45a4a4d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-23T01:04:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 45a4a4d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-23T01:13:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 45a4a4d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-24T02:40:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: f4da2ad
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-24T03:36:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: f4da2ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-24T03:36:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: f4da2ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-24T04:03:36Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: f4da2ad
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-24T05:21:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: f4da2ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

bencrabtree · 2024-02-26T18:52:04Z

src/sagemaker/jumpstart/cache.py

                self.get_region(),
                self._s3_client
            )
-            model_specs = JumpStartModelSpecs(hub_content.content_document, is_hub_content=True)
+            # TODO: Parse HubContentDescription


We'll actually parse hub_model_description.hub_content_document inside of JumpStartModelSpecs. You can see I stubbed out the code for from_content_document in that data class

src/sagemaker/jumpstart/cache.py

bencrabtree · 2024-02-26T18:54:09Z

src/sagemaker/jumpstart/curated_hub/curated_hub.py

+        self,
+        hub_name: str,
+        region: str = JUMPSTART_DEFAULT_REGION_NAME,
+        sagemaker_session: Optional[Session] = None,


We'll have conflicts in my PR #4439, but we should default to DEFAULT_JUMPSTART_SAGEMAKER_SESSION here

I actually thought about that and decided to derive session from the region if left None.DEFAULT_JUMPSTART_SAGEMAKER_SESSION is basically Session with JUMPSTART_DEFAULT_REGION_NAME which is us-west-2. It is a bit weird to get both region and the session, even with the regions check in line 37. I guess we can derive the region from the sagemaker_session actually.

Sorry just seeing this reply- yes you're totally right. We don't need the region param since we can derive that from session

src/sagemaker/jumpstart/curated_hub/curated_hub.py

bencrabtree · 2024-02-26T18:56:07Z

src/sagemaker/jumpstart/curated_hub/utils.py

+) -> HubContentDescription:
+    """Returns descriptive information about the Hub model."""
+
+    sagemaker_session = Session(boto3.Session(region_name=region))


For both fns in this file, I think we should 1/ accept a custom SM Session and 2/ default to the DEFAULT_JUMPSTART_SAGEMAKER_SESSION

Reason behind not using the DEFAULT_JUMPSTART_SAGEMAKER_SESSION was because Hub is technically not just for JS.

bencrabtree · 2024-02-26T18:56:35Z

src/sagemaker/jumpstart/types.py

    """Enum for Hub data storage objects."""

    HUB = "hub"
    MODEL = "model"
    NOTEBOOK = "notebook"

+    @classmethod
+    @property
+    def content_only(cls):


This is nice!

bencrabtree · 2024-02-26T18:57:58Z

src/sagemaker/jumpstart/types.py

@@ -1001,6 +1007,206 @@ def __init__(
        self.id_info = id_info


+class HubContentDependency(JumpStartDataHolderType):


I actually didn't know we had this in the response :) Curious to see how we can leverage this in the copy workflow

bencrabtree · 2024-02-26T18:58:23Z

src/sagemaker/jumpstart/types.py

+        return json_obj
+
+
+class HubContentDescription(JumpStartDataHolderType):


I think we can just call this HubContent

akozd · 2024-02-26T19:50:52Z

src/sagemaker/jumpstart/types.py

+        self.hub_content_version: str = json_obj["hub_content_version"]
+        self.hub_name: str = json_obj["hub_name"]
+
+    def to_json(self) -> Dict[str, Any]:


nit: can this method be inherited from a parent class? Looks the same as to_json of HubDescription

src/sagemaker/jumpstart/curated_hub/curated_hub.py

mufaddal-rohawala · 2024-02-28T23:02:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dddd0a6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

evakravi · 2024-02-28T23:12:50Z

src/sagemaker/jumpstart/cache.py

-            hub_info = hub.describe()
-            return JumpStartCachedContentValue(formatted_content=hub_info)
+        if data_type == HubType.HUB:
+            hub_name, _, _, _ = hub_utils.get_info_from_hub_resource_arn(id_info)


can u make the return type a data class in the next PR?

Clarifying question: Did you mean something like this?

hub_info: HubInfo = hub_utils.get_info_from_hub_resource_arn(id_info)

yes, exactly!

evakravi · 2024-02-28T23:14:08Z

src/sagemaker/jumpstart/curated_hub/utils.py

+    if bucket_name is None:
+        bucket_name: str = generate_default_hub_bucket_name(sagemaker_session)
+
+    sagemaker_session._create_s3_bucket_if_it_does_not_exist(


you may want to do an ownership check, in case someone snipes the bucket

+1 there is also the question of how we would work around this if the bucket is already taken. Perhaps we say this is very unlikely, but we should at least emit a warn in that case.

aye aye it seems that the base function does have the ownership check. Wanted to try reusing existing functions but I think we just need to write up our version so we can check ownership.

bencrabtree · 2024-02-28T23:12:47Z

src/sagemaker/jumpstart/cache.py

            )
+
+            model_specs = JumpStartModelSpecs(DescribeHubContentsResponse(hub_model_description), is_hub_content=True)


Alternative would be to wrap the response in Session with DescribeHubContentResponse. I'd like to hear @evakravi or @akozd opinion on this

we should do one or the other, and stick with that throughout the session module. Personally I'd prefer returning the class rather than a dictionary

I did make all data classes for HubAPI responses. But it seems that having those as return types in session would cause a circular dependency as we have to import jumpstart.types in session and import session in jumpstart.types, which says it is not a good organization. I will leave the session return types as dictionary and type case somewhere else.

bencrabtree · 2024-02-28T23:14:32Z

src/sagemaker/jumpstart/types.py

+        return json_obj
+
+
+class DescribeHubContentsResponse(JumpStartDataHolderType):


Typo? I think this should be DescribeHubContentResponse, singular

mufaddal-rohawala · 2024-02-28T23:57:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-28T23:58:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T00:28:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T01:56:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: dddd0a6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T05:14:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dddd0a6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T06:02:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T06:09:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T06:29:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2024-02-29T06:31:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: dddd0a6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

whittech1 · 2024-03-06T18:52:22Z

src/sagemaker/jumpstart/cache.py

-            hub = CuratedHub(hub_name=info.hub_name, region=info.region)
-            hub_content = hub.describe_model(
-                model_name=info.hub_content_name, model_version=info.hub_content_version
+            hub_model_description: Dict[str, Any] = self._sagemaker_session.describe_hub_content(


Don't we need an existence check on the session variable since it could be None?

I had self._sagemaker_session to default to DEFAULT_JUMPSTART_SAGEMAKER_SESSION unless it changed... I will take a look in the next PR.

whittech1 · 2024-03-06T18:57:56Z

src/sagemaker/jumpstart/cache.py

            )
+
+            model_specs = JumpStartModelSpecs(DescribeHubContentsResponse(hub_model_description), is_hub_content=True)


Can we keep the constructor for JumpStartModelSpecs as taking just the spec dictionary? The from_hub_content() method could easily be a function outside of that class. This keeps us from going down a route where we may end up violating Single Responsibility Principle on JumpStartModelSpecs.

@whittech1 To clarify, are you pointing out the typecase in

model_specs = JumpStartModelSpecs(DescribeHubContentsResponse(hub_model_description), is_hub_content=True)

and instead it should be:

model_specs = JumpStartModelSpecs(hub_model_description, is_hub_content=True)

?

I am saying that the constructor for JumpStartModelSpecs does not need is_hub_content as a flag. The caller can do:

def convert_hub_content_to_js_model_specs() .... # return a dict that matches `JumpStartModelSpecs` constructor's dictionary .... spec_dict = convert_hub_content_to_js_model_spec(DescribeHubContentsResponse(hub_model_description)) model_specs = JumpStartModelSpecs(spec_dict)

whittech1 · 2024-03-06T19:08:42Z

src/sagemaker/jumpstart/curated_hub/utils.py

+    if bucket_name is None:
+        bucket_name: str = generate_default_hub_bucket_name(sagemaker_session)
+
+    sagemaker_session._create_s3_bucket_if_it_does_not_exist(


+1 there is also the question of how we would work around this if the bucket is already taken. Perhaps we say this is very unlikely, but we should at least emit a warn in that case.

…aws#4448)

* prepare release v2.210.0 * update development version to v2.210.1.dev0 * feat: Add new Triton DLC URIs (#4432) * Add new Triton DLC URIs * Update according to black and pylint * feat: Support selective pipeline execution between function step and regular step (#4392) * feat: Add AutoMLV2 support (#4461) * Add AutoMLV2 support * Improvements of the integration tests --------- Co-authored-by: Anton Repushko <[email protected]> * feature: Add TensorFlow 2.14 image configs (#4446) * fix: remove enable_network_isolation from the python doc (#4465) Co-authored-by: Rohan Gujarathi <[email protected]> * doc: Add doc for new feature processor APIs and classes (#4250) * fix: properly close sagemaker config file after loading config (#4457) Closes #4456 * feat: instance specific jumpstart host requirements (#4397) * feat: instance specific jumpstart host requirements * chore: add js support for copies resource requirement, enforce coupling with ResourceRequirements class * fix: typing * fix: pylint * change: Bump Apache Airflow version to 2.8.2 (#4470) * Update tox.ini * Update test_requirements.txt * fix: make sure gpus are found in local_gpu run (#4384) * fix: make sure gpus are found in local_gpu run * fix: black formatting * fix: adjust unit test * feat: pin dll version to support python3.11 to the sdk (#4472) Co-authored-by: Ashwin Krishna <[email protected]> * fix: Skip No Canvas regions for test_deploy_best_candidate (#4477) * prepare release v2.211.0 * update development version to v2.211.1.dev0 * change: Enhance model builder selection logic to include model size (#4429) * change: Enhance model builder selection logic to include model size * Fix conflicts * Address PR comments * fix formatting * fix formatting of test * Fix token in tasks.json * Increase coverage for tests * fix formatting * Fix requirements * Import code instead of importing accelerate * Fix formatting * Setup dependencies * change: Upgrade smp to version 2.2 (#4479) * upgrading smp to version 2.2 * fixing linting issue * fixing syntax error with multiline if statement * upgrading smp to version 2.2 * fixing linting issue * fixing syntax error with multiline if statement * fixing formatting --------- Co-authored-by: Andrew Tian <[email protected]> * feat: Update SM Python SDK for PT 2.2.0 SM DLC (#4481) * update pt2.2 sm training dlc pysdk * update pt2.2 sm inference dlc pysdk and region list * fix: Create custom tarfile extractall util to fix backward compatibility issue (#4476) * fix: Create custom tarfile extractall util to fix backward compatibility issue * Address review comments * fix logger.error statements * prepare release v2.212.0 * update development version to v2.212.1.dev0 * change: Update tblib constraint (#4452) * fix: make unit tests compatible with pytest-xdist (#4486) * fix: make unit tests compatible with pytest-xdist * fix failing test * feature: Add overriding logic in ModelBuilder when task is provided (#4460) * feat: Add Optional task to Model * Revert "feat: Add Optional task to Model" This reverts commit fd3e86b. * Add override logic in ModelBuilder with task provided * Adjusted formatting * Add extra unit tests for invalid inputs * Address PR comments * Add more test inputs to integration test * Add model_metadata field to ModelBuilder * Update doc * Update doc * Adjust formatting --------- Co-authored-by: Samrudhi Sharma <[email protected]> Co-authored-by: Xiong Zeng <[email protected]> * feature: Accept user-defined env variables for the entry-point (#4175) * fix: Move sagemaker pysdk version check after bootstrap in remote job (#4487) * change: enable github actions for PRs (#4489) * change: enable github actions for PRs * Update codebuild-ci.yml * trigger on pull_request_target * add source-version-override * fix permission * feature: Add ModelDataSource and SourceUri support for model package and while registering (#4492) Co-authored-by: Erick Benitez-Ramos <[email protected]> * feat: support JumpStart proprietary models (#4467) * feat: add proprietary manifest/specs parsing add unittests for test_cache small refactoring address comments and more unittests fix linting and fix more tests fix: pylint feat: JumpStartModel class for prop models * remove unused imports and fix docstyle * fix: remove unused args * fix: remove unused args * fix: more unused vars * fix: slow tests * fix: unittests * added more tests to cover some lines * remove estimator warn check * chore: address comments re performance * fix: address comments * complete list experience and other fixes * fix: pylint * add doc utils and fix pylint * fix: docstyle * fix: doc * fix: default payloads * fix: doc and tags and enums * fix: jumpstart doc * rename to open_weights and fix filtering * update filter name * doc update * fix: black * rename to proprietary model and fix unittests * address comments * fix: docstyle and flake8 * address more comments and fix doc * put back doc utils for future refactoring * add prop model title in doc * doc update --------- Co-authored-by: liujiaor <[email protected]> * chore: emit warning when no instance specific gated training env var is available, and raise exception when accept_eula flag is not supplied (#4485) * fix: raise exception when no instance specific gated training env var available * chore: raise client exception if accept_eula flag is not set for gated models * chore: address flake8 errors * chore: emit warning when instance type is chosen with no gated training artifacts * change: bump jinja2 to 3.1.3 in doc/requirments.txt (#4421) (#4423) * change: bump jinja2 to 3.1.3 in doc/requirments.txt (#4421) * change: bump jinja2 to 3.1.3 in doc/requirments.txt * Update requirements.txt * feature: TGI 1.4.0 (#4424) * documentation: fix the ClarifyCheckStep documentation to mention PDP (#4259) * documentation: fix the ClarifyCheckStep documentation to mention PDP support * fix: break the lines to meet pylint requirement --------- Co-authored-by: Shing Lyu <[email protected]> * documentation: Explain the ClarifyCheckStep and QualityCheckStep parameters (#4261) * documentation: explain the ClarifyCheckStep and QualityCheckStep parameters * fix: remove trailing space --------- Co-authored-by: Shing Lyu <[email protected]> * feat: Telemetry metrics (#4414) * Emit additional telemetry metrics * Fix unit tests * Emit endpoint failure to telemetry * Address PR Comments * Emit latency in telemetry * Address PR Comments * Addressed PR Comments * Address PR Comments * Fix tests * Fix integ tests --------- Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> * documentation: change order of pipelines topics (#4427) * prepare release v2.208.0 * update development version to v2.208.1.dev0 * feature: AutoGluon 1.0.0 image_uris update (#4426) --------- Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: Jinyoung Lim <[email protected]> Co-authored-by: Shing Lyu <[email protected]> Co-authored-by: Shing Lyu <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: stacicho <[email protected]> Co-authored-by: ci <ci> Co-authored-by: tonyhu <[email protected]> * feat: add hub and hubcontent support in retrieval function for jumpstart model cache (#4438) * feat: jsch jumpstart estimator support (#4439) * Master jumpstart curated hub (#4464) * add hub_arn support for accept_types, content_types, serializers, deserializers, and predictor (#4463) * feature: JumpStart CuratedHub class creation and function definitions (#4448) * MultiPartCopy with Sync Algorithm (#4475) * first pass at sync function with util classes * adding tests and update clases * linting * file generator class inheritance * lint * multipart copy and algorithm updates * modularize sync * reformatting folders * testing for sync * do not tolerate vulnerable * remove prints * handle multithreading progress bar * update tests * optimize function and add hub bucket prefix * docstrings and linting * rebase with master * bad rebase * trying to fix codecov * uncomment codebuild-ci --------- Co-authored-by: ci <ci> Co-authored-by: Nikhil Kulkarni <[email protected]> Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Anton Repushko <[email protected]> Co-authored-by: Anton Repushko <[email protected]> Co-authored-by: Sai Parthasarathy Miduthuri <[email protected]> Co-authored-by: Rohan Gujarathi <[email protected]> Co-authored-by: Rohan Gujarathi <[email protected]> Co-authored-by: cansun <[email protected]> Co-authored-by: Justin <[email protected]> Co-authored-by: evakravi <[email protected]> Co-authored-by: Kalyani Nikure <[email protected]> Co-authored-by: gv <[email protected]> Co-authored-by: akrishna1995 <[email protected]> Co-authored-by: Ashwin Krishna <[email protected]> Co-authored-by: Samrudhi Sharma <[email protected]> Co-authored-by: adtian2 <[email protected]> Co-authored-by: Andrew Tian <[email protected]> Co-authored-by: Sirut Buasai <[email protected]> Co-authored-by: Danny Bushkanets <[email protected]> Co-authored-by: Erick Benitez-Ramos <[email protected]> Co-authored-by: xiongz945 <[email protected]> Co-authored-by: Samrudhi Sharma <[email protected]> Co-authored-by: Xiong Zeng <[email protected]> Co-authored-by: martinRenou <[email protected]> Co-authored-by: mrudulmn <[email protected]> Co-authored-by: Haotian An <[email protected]> Co-authored-by: liujiaor <[email protected]> Co-authored-by: Jinyoung Lim <[email protected]> Co-authored-by: Shing Lyu <[email protected]> Co-authored-by: Shing Lyu <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: Jonathan Makunga <[email protected]> Co-authored-by: stacicho <[email protected]> Co-authored-by: tonyhu <[email protected]>

…aws#4448)

* fix: Move sagemaker pysdk version check after bootstrap in remote job (#4487) * feat: support JumpStart proprietary models (#4467) * feat: add proprietary manifest/specs parsing add unittests for test_cache small refactoring address comments and more unittests fix linting and fix more tests fix: pylint feat: JumpStartModel class for prop models * remove unused imports and fix docstyle * fix: remove unused args * fix: remove unused args * fix: more unused vars * fix: slow tests * fix: unittests * added more tests to cover some lines * remove estimator warn check * chore: address comments re performance * fix: address comments * complete list experience and other fixes * fix: pylint * add doc utils and fix pylint * fix: docstyle * fix: doc * fix: default payloads * fix: doc and tags and enums * fix: jumpstart doc * rename to open_weights and fix filtering * update filter name * doc update * fix: black * rename to proprietary model and fix unittests * address comments * fix: docstyle and flake8 * address more comments and fix doc * put back doc utils for future refactoring * add prop model title in doc * doc update --------- Co-authored-by: liujiaor <[email protected]> * feat: add hub and hubcontent support in retrieval function for jumpstart model cache (#4438) * feat: jsch jumpstart estimator support (#4439) * Master jumpstart curated hub (#4464) * add hub_arn support for accept_types, content_types, serializers, deserializers, and predictor (#4463) * feature: JumpStart CuratedHub class creation and function definitions (#4448) * MultiPartCopy with Sync Algorithm (#4475) * first pass at sync function with util classes * adding tests and update clases * linting * file generator class inheritance * lint * multipart copy and algorithm updates * modularize sync * reformatting folders * testing for sync * do not tolerate vulnerable * remove prints * handle multithreading progress bar * update tests * optimize function and add hub bucket prefix * docstrings and linting * rebase with master * bad rebase * support for gated and training unsupported * merge with master-curated-jumpstart * linting * update types * update * update bootstrap * fix codecov --------- Co-authored-by: qidewenwhen <[email protected]> Co-authored-by: Haotian An <[email protected]> Co-authored-by: liujiaor <[email protected]> Co-authored-by: Jinyoung Lim <[email protected]>

…aws#4448)

initial barebone for hub utils and curated hub

4922511

jinyoung-lim requested a review from a team as a code owner February 22, 2024 21:25

jinyoung-lim requested review from mohanasudhan and removed request for a team February 22, 2024 21:25

evakravi reviewed Feb 22, 2024

View reviewed changes

bencrabtree reviewed Feb 22, 2024

View reviewed changes

refactor, add types for hub/hubcontent descriptions, add helpers.

f4da2ad

jinyoung-lim force-pushed the feature/hubutil branch from 45a4a4d to f4da2ad Compare February 24, 2024 02:34

bencrabtree reviewed Feb 26, 2024

View reviewed changes

akozd reviewed Feb 26, 2024

View reviewed changes

jinyoung-lim commented Feb 26, 2024

View reviewed changes

src/sagemaker/jumpstart/curated_hub/curated_hub.py Outdated Show resolved Hide resolved

Merge branch 'master-jumpstart-curated-hub' into feature/hubutil

d5937b3

jinyoung-lim added 3 commits February 28, 2024 22:51

Use sagemaker.session.Session to call HubAPIs in cache

0937c74

Merge branch 'master-jumpstart-curated-hub' into feature/hubutil

7c87c52

lint

dddd0a6

evakravi reviewed Feb 28, 2024

View reviewed changes

bencrabtree approved these changes Feb 28, 2024

View reviewed changes

evakravi approved these changes Feb 28, 2024

View reviewed changes

evakravi merged commit 352a5c1 into aws:master-jumpstart-curated-hub Feb 29, 2024

whittech1 reviewed Mar 6, 2024

View reviewed changes

jinyoung-lim added a commit to jinyoung-lim/sagemaker-python-sdk that referenced this pull request Mar 8, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

386a291

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 13, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

30f4e90

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 13, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

52eae82

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 18, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

82d0d92

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 20, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

6d6c170

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 21, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

f4c72ca

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 23, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

1c0b8b3

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 23, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

36080da

…aws#4448)

bencrabtree pushed a commit to bencrabtree/sagemaker-python-sdk that referenced this pull request Mar 23, 2024

feature: JumpStart CuratedHub class creation and function definitions (…

7ec4828

…aws#4448)

		return f"sagemaker-hubs-{region}-{account_id}"


		def create_hub(

		@@ -1001,6 +1007,206 @@ def __init__(
		self.id_info = id_info


		class HubContentDependency(JumpStartDataHolderType):

		return json_obj


		class HubContentDescription(JumpStartDataHolderType):

		)

		model_specs = JumpStartModelSpecs(DescribeHubContentsResponse(hub_model_description), is_hub_content=True)

		return json_obj


		class DescribeHubContentsResponse(JumpStartDataHolderType):

feature: JumpStart CuratedHub class creation and function definitions #4448

feature: JumpStart CuratedHub class creation and function definitions #4448

Conversation

jinyoung-lim commented Feb 22, 2024 • edited Loading

Merge Checklist

General

Tests

mufaddal-rohawala commented Feb 22, 2024

AWS CodeBuild CI Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bencrabtree Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mufaddal-rohawala commented Feb 22, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 22, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 22, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 22, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 23, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 23, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 23, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 23, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 23, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 23, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 24, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 24, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 24, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 24, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 24, 2024

AWS CodeBuild CI Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mufaddal-rohawala commented Feb 28, 2024

AWS CodeBuild CI Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mufaddal-rohawala commented Feb 28, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 28, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 29, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 29, 2024

AWS CodeBuild CI Report

mufaddal-rohawala commented Feb 29, 2024

AWS CodeBuild CI Report

jinyoung-lim commented Feb 22, 2024 •

edited

Loading

bencrabtree Feb 22, 2024 •

edited

Loading

whittech1 Mar 6, 2024 •

edited

Loading