entry point not detected in Pytorch script mode when in source folder

[hjuhel-cdpq]

Describe the bug
While using:

• A custom image, forked from the Pytorch official image
• Some custom training code, located in the code folder (code/train.py),

The entrypoint train.py is never detected

Expected behavior
As stated in the documentation, the entry point schould be correctly detected

System information

• SageMaker Python SDK version:
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): Pytorch
• Framework version: 1.4.0
• Python version: 3.6
• CPU or GPU: GPU
• Custom Docker image (Y/N): Y

To reproduce

1. Create a docker image, adding requirements to the official pytorch image and push it to ECR

FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.4.0-gpu-py36-cu101-ubuntu16.04

## Install and Download Spacy models
RUN pip install spacy
RUN python -m spacy download "en"
RUN python -m spacy download "en_core_web_lg"
RUN python -m spacy download "en_core_web_md"

2. In Jupyter sagemaker, create a folder code containing the following train.py script:

def train(args):
    print("done")

if __name__ == "__main__":

    # Retrieve hyperparameters
    parser = argparse.ArgumentParser()

    # Parse the hyperparameters
    parser.add_argument("--embedding_dim", type=int, default=300)

    args = parser.parse_args()

    try:
        _train(args)
        sys.exit(1)

    else:
        sys.exit(0)

The folder tree must looks like :

$ tree code
code
|-- train.py

3. Instanciate a Pytorch training job with :

Please, note that source_dir is the folder containing the entrypoint :

import sagemaker
from sagemaker.pytorch import PyTorch
from sagemaker.estimator import Estimator
from sagemaker import get_execution_role

ROLE = get_execution_role()
INSTANCE = 'ml.p3.2xlarge'

output_path = '<path_to_s3_location>'

pytorch_estimator = PyTorch(entry_point='train.py',
                            source_dir="code",
                            image_name="<path_to_docker_image>",
                            base_job_name="foobar",
                            train_instance_type=INSTANCE,
                            train_instance_count=1,
                            role = ROLE
                           )

pytorch_estimator.fit()

4. The job fail with error :

/opt/conda/bin/python train.py

/opt/conda/bin/python: can't open file 'train.py': [Errno 2] No such file or directory
2020-03-04 18:05:55,332 sagemaker-containers ERROR    ExecuteUserScriptError:
Command "/opt/conda/bin/python train.py"
/opt/conda/bin/python: can't open file 'train.py': [Errno 2] No such file or directory

Screenshots or logs
Complete log is :

No framework_version specified, defaulting to version 0.4. This is not the latest supported version. If you would like to use version 1.3.1, please add framework_version=1.3.1 to your constructor.
2020-03-04 18:00:29 Starting - Starting the training job...
2020-03-04 18:00:30 Starting - Launching requested ML instances......
2020-03-04 18:01:35 Starting - Preparing the instances for training......
2020-03-04 18:02:45 Downloading - Downloading input data
2020-03-04 18:02:45 Training - Downloading the training image..................
2020-03-04 18:05:56 Uploading - Uploading generated training modelbash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2020-03-04 18:05:52,689 sagemaker-containers INFO     Imported framework sagemaker_pytorch_container.training
2020-03-04 18:05:52,713 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2020-03-04 18:05:52,714 sagemaker_pytorch_container.training INFO     Invoking user training script.
2020-03-04 18:05:53,164 sagemaker-containers INFO     Module default_user_module_name does not provide a setup.py. 
Generating setup.py
2020-03-04 18:05:53,164 sagemaker-containers INFO     Generating setup.cfg
2020-03-04 18:05:53,164 sagemaker-containers INFO     Generating MANIFEST.in
2020-03-04 18:05:53,165 sagemaker-containers INFO     Installing module with the following command:
/opt/conda/bin/python -m pip install . 
Processing /tmp/tmpzvar_c2z/module_dir
Building wheels for collected packages: default-user-module-name
  Building wheel for default-user-module-name (setup.py): started
  Building wheel for default-user-module-name (setup.py): finished with status 'done'
  Created wheel for default-user-module-name: filename=default_user_module_name-1.0.0-py2.py3-none-any.whl size=4364 sha256=cbf38ce610a10a5f5588edd12c6984667076665eaadbcfb858505e580ff243a5
  Stored in directory: /tmp/pip-ephem-wheel-cache-lfxv8mpn/wheels/52/eb/a1/d0d0d1552198986736428c8139b9ba7e9d91768a9bfb9b8094
Successfully built default-user-module-name
Installing collected packages: default-user-module-name
Successfully installed default-user-module-name-1.0.0
2020-03-04 18:05:55,303 sagemaker-containers INFO     Invoking user script

Training Env:

{
    "additional_framework_parameters": {},
    "channel_input_dirs": {
        "training": "/opt/ml/input/data/training"
    },
    "current_host": "algo-1",
    "framework_module": "sagemaker_pytorch_container.training:main",
    "hosts": [
        "algo-1"
    ],
    "hyperparameters": {},
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {
        "training": {
            "TrainingInputMode": "File",
            "S3DistributionType": "FullyReplicated",
            "RecordWrapperType": "None"
        }
    },
    "input_dir": "/opt/ml/input",
    "is_master": true,
    "job_name": "sentiment10k-sgmk-cnn-2-2020-03-04-18-00-28-905",
    "log_level": 20,
    "master_hostname": "algo-1",
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://leo-data/sentiment10k-sgmk-cnn-2-2020-03-04-18-00-28-905/source/sourcedir.tar.gz",
    "module_name": "/opt/ml/code/train",
    "network_interface_name": "eth0",
    "num_cpus": 8,
    "num_gpus": 1,
    "output_data_dir": "/opt/ml/output/data",
    "output_dir": "/opt/ml/output",
    "output_intermediate_dir": "/opt/ml/output/intermediate",
    "resource_config": {
        "current_host": "algo-1",
        "hosts": [
            "algo-1"
        ],
        "network_interface_name": "eth0"
    },
    "user_entry_point": "/opt/ml/code/train.py"
}

Environment variables:

SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={}
SM_USER_ENTRY_POINT=/opt/ml/code/train.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=["training"]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=/opt/ml/code/train
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://.../source/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"...","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://.../source/sourcedir.tar.gz","module_name":"/opt/ml/code/train","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"},"user_entry_point":"/opt/ml/code/train.py"}
SM_USER_ARGS=[]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TRAINING=/opt/ml/input/data/training
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages

Invoking script with the following command:

/opt/conda/bin/python train.py

/opt/conda/bin/python: can't open file 'train.py': [Errno 2] No such file or directory
2020-03-04 18:05:55,332 sagemaker-containers ERROR    ExecuteUserScriptError:
Command "/opt/conda/bin/python train.py"
/opt/conda/bin/python: can't open file 'train.py': [Errno 2] No such file or directory

Could you please help moi figuring what I'm doing wrong ?

[ehsanmok]

Examine your SAGEMAKER_PROGRAM env variable and see what's missing.

General development tips:

1. Local mode
2. Enable logs in debug mode use container_log_level=10.

[hjuhel-cdpq]

Thanks @ehsanmok for your answer.

The SAGEMAKER_PROGRAM was correctly set-up. The issue was due to a requirements.txt. It's unclear for me, why was happening, but copying the requirements to /opt/ml/code was, somehow, causing the bug

Switching from

COPY requirements.txt /opt/ml/code/requirements.txt
RUN pip install /opt/ml/code/requirements.txt

to :

COPY requirements.txt /home/requirements.txt
RUN pip install /home/requirements.txt

resolved the issue.

[LecJackS]

Just for the record, this also happened to me while cloning a repo (with a requirements.txt file) inside /opt/ml/code

Cloning it outside /code (in /opt/ml/) solved the problem for me.