SageMaker Framework Processing step not finding /opt/ml/processing/input/code/

[vickytang1993]

Hi,

I'm using FrameworkProcessor from the SageMaker Python SDK for a ProcessingStep in my SageMaker pipeline. When running the pipeline from a Jupyter notebook in SageMaker Studio, I'm getting the following error:

2023-11-25T19:16:07.921+05:30Copy  File "/opt/ml/processing/input/entrypoint/runproc.sh", line 3    
cd /opt/ml/processing/input/code/ ^


2023-11-25T19:16:07.921+05:30CopySyntaxError: invalid syntax

This is from the script runproc.sh, which is generated by FrameworkProcessor. It looks like the script is trying to go to the directory "/opt/ml/processing/input/code/" to find the entrypoint python file to run for the processing but can't find the file. Here is my Python code for my pipeline:

from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker import get_execution_role
from sagemaker.processing import FrameworkProcessor
import sagemaker.processing as sm_processing

# # Create a FrameworkProcessorArgs object
# framework_processor_args = sm_processing.FrameworkProcessorArgs(
#     framework_entrypoint_command=["Python"]
# )
from sagemaker.workflow.pipeline_context import PipelineSession

session = PipelineSession()


image_uri = "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.8-cpu-py3"
s3_bucket ='di-dev-sagemaker'
inference_output = f"s3://{s3_bucket}/Flanges-Corrosion/BatchTransformOutputJson"
input_json_dir = "Flanges-Corrosion/BatchTransformOutputJson"
script_eval = FrameworkProcessor(
    framework_version='1.8',
    estimator_cls=PyTorch,
    image_uri=image_uri,
    
    instance_type="ml.m5.xlarge",
    instance_count=1,
    base_job_name="job-args-processor",
    role=role,
    sagemaker_session = session

)
script_eval.framework_entrypoint_command= ['python3']
# Run the processing job
step_args = script_eval.run(
    code="pp_main.py",
    # code = "entrypoint.sh",
    
    source_dir='VT_postprocessing',
    inputs= None,
    # [
    #         ProcessingInput(source="VT_postprocessing/data_handler.py", destination="/opt/ml/processing/input/code/"),
    #         ],
  
    outputs= None
   
)




step_postprocess = ProcessingStep(
    name="Framework-PostProcessing-v1",
    # processor=script_eval,
    
    inputs = None,
    # [
    #         ProcessingInput(source="VT_postprocessing/data_handler.py", destination="/opt/ml/processing/input/code/data_handler.py"),
    #         ],
    outputs = None,
    # code="VT_postprocessing/pp_main.py",
    job_arguments = ["--s3_bucket",s3_bucket,"--input_json_dir",input_json_dir],
    step_args = step_args
)
step_postprocess.add_depends_on([step_transform_corr_seg, step_transform_langsam])

I would appreciate any help with this. I found an issue regarding the broken integration between FrameworkProcessor and ProcessingStep (#2909). Is it related?

[trungleduc]

Hi @vickytang1993 , could you provide a fully reproducible code (including external code, and data...)?

[martinRenou]

Closing for triaging. Feel free to continue the discussion or reopen the issue if needed.

Thanks!