The method _run_clone_command does not support git/CodeCommit ssh URLs

[jasm37]

System Information

• Framework: PyTorch
• Framework Version: 1.1.0
• Python Version: 3.7
• CPU or GPU: GPU
• Python SDK Version: 1.43.2
• Are you using a custom image: No

Describe the problem

Basically the method _run_clone_command in sagemaker/git_utils.py does not seem to handle a repo_url input that starts with ssh://... I found this while trying to run a hyperparameter tuning job with a PyTorch estimator with git_config=ssh://git-codecommit.us-east-2.amazonaws.com/v1/repos/my_repo.

Minimal repro / logs

Traceback (most recent call last):
  File "/home/user/code/sagemaker_runner/main.py", line 65, in <module>
    tuner.fit()
  File "/home/user/anaconda3/envs/aws/lib/python3.7/site-packages/sagemaker/tuner.py", line 359, in fit
    self.estimator._prepare_for_training(job_name)
  File "/home/user/anaconda3/envs/aws/lib/python3.7/site-packages/sagemaker/estimator.py", line 1431, in _prepare_for_training
    self.git_config, self.entry_point, self.source_dir, self.dependencies
  File "/home/user/anaconda3/envs/aws/lib/python3.7/site-packages/sagemaker/git_utils.py", line 101, in git_clone_repo
    raise ValueError("Source directory does not exist in the repo.")
ValueError: Source directory does not exist in the repo.

All the following methods are located in sagemaker/git_utils.py.
This is my explanation as of why this happens:
When the method git_clone_repo is called, the repo seems to not be cloned because the if/elif branching in _run_clone_command (called deep inside _generate_and_run_clone_command) does not support the case where repo_url starts with ssh. Then the source directory cannot be found because the temp directory where the repository is supposed to be is empty. Is this correct or am I missing something here?

• Exact command to reproduce:

...
git_config = {'repo': 'ssh://git-codecommit.us-east-2.amazonaws.com/v1/repos/my_repo'}
estimator = PyTorch(entry_point='aws_main.py',
                    role=role,
                    framework_version='1.1.0',
                    train_instance_count=1,
                    train_instance_type=instance_type,
                    source_dir="run/",
                    git_config=git_config)
...
...
tuner = HyperparameterTuner(
                            estimator=estimator,
                            objective_metric_name=metric_name,
                            hyperparameter_ranges=hyperparameter_ranges,
                            metric_definitions=metric_definitions,
                            objective_type='Minimize',
                            max_jobs=max_jobs,
                            max_parallel_jobs=max_parallel_jobs,
                            base_tuning_job_name="my-repo-param-tuning"
                            )
tuner.fit()

[laurenyu]

Thanks for bringing this to our attention! I think you're right that this is a bug - as seen here and here, the code assumes that an SSH URL starts with "git@" but doesn't include the possibility of "ssh://".

[akrishna1995]

Thank you for raising this issue! Unfortunately, this example notebook no longer exists. You can find our most up-to-date Reinforcemnet Learning example notebooks here: https://github.com/aws/amazon-sagemaker-examples/tree/main/reinforcement_learning. Please feel free to reopen if the issue still persists with the latest sagemaker