Skip to content

JumpStart Uncompressed Format Causing AttributeError for JumpStart ID Local Container Mode #4283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gwang111 opened this issue Nov 30, 2023 · 1 comment · Fixed by #4294
Closed

Comments

@gwang111
Copy link
Collaborator

Describe the bug
When building a JumpStart ID model with ModelBuilder, an AttributeError occurs when fetching the JumpStart resources from S3

To reproduce

model_builder = ModelBuilder(
    model="huggingface-llm-falcon-7b-bf16",
    schema_builder=SchemaBuilder(sample_input, sample_output),
    model_path="/path",
    mode=Mode.LOCAL_CONTAINER
)

builder = model_builder.build()

Expected behavior
The uncompressed model resources should be fetched from S3

Screenshots or logs

INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml
INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole
WARNING:sagemaker.jumpstart:Using model 'huggingface-llm-falcon-7b-bf16' with wildcard version identifier '*'. You can pin to version '2.0.0' for more stable results. Note that models may have different input/output signatures after a major version upgrade.
ModelBuilder: INFO:     JumpStart Model ID detected.
INFO:sagemaker.jumpstart:No instance type selected for inference hosting endpoint. Defaulting to ml.g5.2xlarge.
ModelBuilder: INFO:     JumpStart ID huggingface-llm-falcon-7b-bf16 is packaged with Image URI: 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.1.0-gpu-py39-cu118-ubuntu20.04
ModelBuilder: INFO:     Building for TGI JumpStart Model ID...
ModelBuilder: WARNING:     75.98389809091147 percent of docker disk space at /var/lib/docker is used. Please consider freeing up disk space or increasing the EBS volume if you are on a SageMaker Notebook.
ModelBuilder: INFO:     Downloading JumpStart artifacts from S3...
{'S3DataSource': {'S3Uri': 's3://jumpstart-cache-prod-us-east-1/huggingface-llm/huggingface-llm-falcon-7b-bf16/artifacts/inference-prepack/v1.0.1/', 'S3DataType': 'S3Prefix', 'CompressionType': 'None'}}
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File <timed exec>:1

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/serve/builder/model_builder.py:579, in ModelBuilder.build(self, mode, role_arn, sagemaker_session)
    577 if isinstance(self.model, str):
    578     if self._is_jumpstart_model_id():
--> 579         return self._build_for_jumpstart()
    580     if self._is_djl():
    581         return self._build_for_djl()

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/serve/builder/jumpstart_builder.py:261, in JumpStart._build_for_jumpstart(self)
    258     self.pysdk_model = pysdk_model
    259     self.image_uri = self.pysdk_model.image_uri
--> 261     self._build_for_tgi_jumpstart()
    262 else:
    263     raise ValueError(
    264         "JumpStart Model ID was not packaged with djl-inference or tgi-inference container."
    265     )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/serve/builder/jumpstart_builder.py:222, in JumpStart._build_for_tgi_jumpstart(self)
    220 if self.mode == Mode.LOCAL_CONTAINER:
    221     if not hasattr(self, "prepared_for_tgi"):
--> 222         self.prepared_for_tgi = prepare_tgi_js_resources(
    223             model_path=self.model_path,
    224             js_id=self.model,
    225             dependencies=self.dependencies,
    226             model_data=self.pysdk_model.model_data,
    227         )
    228     self._prepare_for_mode()
    229 elif self.mode == Mode.SAGEMAKER_ENDPOINT and hasattr(self, "prepared_for_tgi"):

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/serve/model_server/tgi/prepare.py:74, in prepare_tgi_js_resources(model_path, js_id, shared_libs, dependencies, model_data)
     58 """Prepare serving when a JumpStart model id is given
     59 
     60 Args:
   (...)
     70 
     71 """
     72 model_path, code_dir = _create_dir_structure(model_path)
---> 74 return _copy_jumpstart_artifacts(model_data, js_id, code_dir)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/serve/model_server/tgi/prepare.py:22, in _copy_jumpstart_artifacts(model_data, js_id, code_dir)
     19 with _tmpdir(directory=str(code_dir)) as js_model_dir:
     20     # TODO: remove if block after 10/30 once everything is shifted to uncompressed
     21     print(model_data)
---> 22     if model_data.endswith("tar.gz"):
     23         subprocess.run(["aws", "s3", "cp", model_data, js_model_dir])
     25         logger.info("Uncompressing JumpStart artifacts for faster loading...")

AttributeError: 'dict' object has no attribute 'endswith'

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.199
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): TGI
  • Framework version: Latest
  • Python version: 3.10
  • CPU or GPU: GPU
  • Custom Docker image (Y/N): N

Additional context
Add any other context about the problem here.

@gwang111
Copy link
Collaborator Author

Fix Here: #4281

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant