Skip to content

Sagemaker SDK on lambda function invoking serverless endpoint #4123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
KaramRazooq opened this issue Sep 16, 2023 · 8 comments · Fixed by #4268
Closed

Sagemaker SDK on lambda function invoking serverless endpoint #4123

KaramRazooq opened this issue Sep 16, 2023 · 8 comments · Fixed by #4268

Comments

@KaramRazooq
Copy link

I'm getting the following message while running the Sagemaker SDK on my lambda function.
I have installed the correct libraries that are comparable with python 3.11 and the lambda.

The log data from lambda:

sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
--
sagemaker.config INFO - Not applying SDK defaults from location: /home/sbx_user1051/.config/sagemaker/config.yaml

The log from the endpoint


2023-09-16T21:55:46,174 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - Executing input_fn from inference.py ...
--
2023-09-16T21:55:46,174 [INFO ] W-9000-model_1.0 org.pytorch.serve.wlm.WorkerThread - Backend response time: 3557
2023-09-16T21:55:46,174 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - Executing predict_fn from inference.py ...
2023-09-16T21:55:46,174 [INFO ] W-9000-model_1.0-stdout MODEL_LOG - Executing output_fn from inference.py ...
2023-09-16T21:55:46,174 [INFO ] W-9000-model_1.0 ACCESS_LOG - /127.0.0.1:35074 "POST /invocations HTTP/1.1" 200 3558
2023-09-16T21:55:46,174 [INFO ] W-9000-model_1.0 TS_METRICS - Requests2XX.Count:1\|#Level:Host\|#hostname:169.254.29.125,timestamp:1694901018

My code is

from sagemaker.pytorch.model import PyTorchModel, PyTorchPredictor 
from sagemaker.deserializers import JSONDeserializer
    
predictor = PyTorchPredictor(endpoint_name=ENDPOINT_NAME, deserializer=JSONDeserializer())
result = predictor.predict(decode)

The predictor function is what's breaking the run and sending the INFO level log

any thoughts on how to fix this?
I tried to look into documentation on what could fix this but did not find anything

@dongreenberg
Copy link

I'm getting this as well, running locally on a Mac.

INFO | 2023-09-21 20:29:41.200081 | Found credentials in shared credentials file: ~/.aws/credentials
sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /Users/donny/Library/Application Support/sagemaker/config.yaml

@RistovaIvona
Copy link

I am also experiencing this issue while running the SageMaker SDK. Has anyone found a solution to this problem?

sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml

@djwbamboo
Copy link

From what I'm observing it seems to be a be associated with versions >=2.183.0. I solved by downgrading:

pip install sagemaker==2.182.0

@ConsciousCoder07
Copy link

Thanks @djwbamboo, that worked for me

@brifordwylie
Copy link
Contributor

My logs were full of these messages....IMO they should make this a debug message instead of info :)

For now just pop this into your code and it should suppress the messages

# Sagemaker continuously complains about config, so we'll suppress it
logging.getLogger("sagemaker.config").setLevel(logging.WARNING)

@stiebels
Copy link

It's extremely annoying. Needed to disable this at various places to avoid cluttering logs.

@bouachalazhar
Copy link

From what I'm observing it seems to be a be associated with versions >=2.183.0. I solved by downgrading:

pip install sagemaker==2.182.0

If I can't do it what can I do ?

@cldixon
Copy link

cldixon commented Sep 30, 2024

My logs were full of these messages....IMO they should make this a debug message instead of info :)

For now just pop this into your code and it should suppress the messages

# Sagemaker continuously complains about config, so we'll suppress it
logging.getLogger("sagemaker.config").setLevel(logging.WARNING)

This worked for me, but I had to place the line before import sagemaker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants