Remote Training Job created with decorator not installing pip dependencies

[asarasua]

Describe the bug
After upgrading to version 2.212.0, pip dependencies passed as a requirements.txt file in @remote decorator are not installed. Same code works fine in versions up to 2.209.x, fails with versions from 2.210.x.

To reproduce
Just tried with the toy example from the documentation:

import boto3
import sagemaker
from sagemaker.remote_function import remote

sm_session = sagemaker.Session(boto_session=boto3.session.Session(region_name="us-east-1"))
settings = dict(
    sagemaker_session=sm_session,
    role=<The IAM role name>,
    instance_type="ml.m5.xlarge",
    dependencies='./requirements.txt'
)

@remote(**settings)
def divide(x, y):
    return x / y


if __name__ == "__main__":
    print(divide(2, 3.0))

And got this in the Training Job, showing no pip install logging before (sagemaker is listed in requirements.txt):

  File "/opt/ml/input/data/sagemaker_remote_function_bootstrap/bootstrap_runtime_environment.py", line 24, in <module>
    from runtime_environment_manager import (
  File "/opt/ml/input/data/sagemaker_remote_function_bootstrap/runtime_environment_manager.py", line 27, in <module>
    import sagemaker
ModuleNotFoundError: No module named 'sagemaker'

Using sagemaker sdk versions previous to 2.210.x, the job prints pip install logging and then runs the operation without problems.

We observed equivalent problems in our code for remote functions that were working fine before upgrading.

Expected behavior
The training job should start by installing the dependencies defined in requirements.txt

System information

• SageMaker Python SDK version: tried with 2.210.0, 2.211.0, 2.212.0
• Python version: 3.8.18
• CPU or GPU: CPU

[nmadan]

Hi @asarasua, thanks for opening an issue for this. We are actively working on a fix.

[arun-akur8]

Hello!

It seems I still get the same error when using remote decorator in VSCode in Code Editor Space in AWS Sagemaker Studio. Would it be the same issue?

Error is :

/miniconda3/bin/python: Error while finding module specification for 'sagemaker.remote_function.invoke_function' (ModuleNotFoundError: No module named 'sagemaker')

Minimal code example:

from sagemaker.remote_function import remote
import sagemaker
import boto3
boto_session = boto3.Session()
sm_session = sagemaker.Session(boto_session,default_bucket="default_bucket")
image_uri = sagemaker.image_uris.retrieve(sagemaker.image_uris.SKLEARN_FRAMEWORK,version="1.2-1",region=region)
settings = dict(
         sagemaker_session=sm_session,
         role=sagemaker.get_execution_role(),
         instance_type="ml.m5.large", image_uri=image_uri)

sagemaker version: 2.200.0, 2.221.0
Python version: 3.8
Boto version: 1.34.109