Cannot use SSH URL to point to a private GitHub repo containing a custom inference script when deploying a model

[e13h]

The SageMaker Python SDK docs imply that you can use your own ssh/config on a notebook instance

For GitHub and other Git repos, when SSH URLs are provided, it doesn’t matter whether 2FA is enabled or disabled; you should either have no passphrase for the SSH key pairs, or have the ssh-agent configured so that you will not be prompted for SSH passphrase when you do ‘git clone’ command with SSH URLs.

But, the code for cloning repos with an SSH URL uses the GIT_SSH environment variable (to make sure that the user isn't prompted for credentials)

sagemaker-python-sdk/src/sagemaker/git_utils.py

Lines 280 to 287 in 9f399ce

	elif repo_url.startswith("git@"):
	with tempfile.NamedTemporaryFile() as sshnoprompt:
	write_pipe = open(sshnoprompt.name, "w")
	write_pipe.write("ssh -oBatchMode=yes $@")
	write_pipe.close()
	os.chmod(sshnoprompt.name, 0o511)
	my_env["GIT_SSH"] = sshnoprompt.name
	subprocess.check_call(["git", "clone", repo_url, dest_dir], env=my_env)

The documentation for git says, however, that whatever GIT_SSH points to will override any ssh/config you have set up.

GIT_SSH, if specified, is a program that is invoked instead of ssh when Git tries to connect to an SSH host. It is invoked like $GIT_SSH [username@]host [-p <port>] <command>. Note that this isn’t the easiest way to customize how ssh is invoked; it won’t support extra command-line parameters, so you’d have to write a wrapper script and set GIT_SSH to point to it. It’s probably easier just to use the ~/.ssh/config file for that.

So this means that you can't clone a private repo using an SSH URL. The only way to clone it would be to use a GitHub Personal Access Token (PAT) and an HTTPS URL.

---

To reproduce, use this template code (having an ssh/config and a private key set up or not makes no difference in the output behavior):

from sagemaker import get_execution_role
from sagemaker.pytorch.model import PyTorchModel

PyTorchModel(
    model_data=s3_uri_path_to_model_tar_gz,
    git_config={
        "repo": f"[email protected]:{repo_account_name}/{private_repo_name}.git",
        "branch": branch_name,
    },
    code_location=s3_uri_path_to_code_location,
    source_dir=path_to_src_in_github_repo,
    entry_point="inference.py",
    role=get_execution_role(),
    py_version="py3",
    framework_version="1.4"
)

Output (not a very helpful message, but that's a different issue 😉 )

CalledProcessError: Command '['git', 'clone', '[email protected]:<org>/<repo>.git', '/tmp/tmp01e7i6rz']' returned non-zero exit status 128.

[timorkal]

Getting the same thing with this logs:

Cloning into '/tmp/tmp9kr9rdwo'...
fatal: cannot exec '/tmp/tmp76jhcf6i': Text file busy
fatal: cannot exec '/tmp/tmp76jhcf6i': Text file busy
fatal: unable to fork
Traceback (most recent call last):
File "sagemaker_edgify_run.py", line 122, in
main()
File "sagemaker_edgify_run.py", line 115, in main
logs=args_dict["logs"]
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/estimator.py", line 677, in fit
self._prepare_for_training(job_name=job_name)
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/estimator.py", line 2206, in _prepare_for_training
self.git_config, self.entry_point, self.source_dir, self.dependencies
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/git_utils.py", line 90, in git_clone_repo
_generate_and_run_clone_command(git_config, dest_dir)
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/git_utils.py", line 159, in _generate_and_run_clone_command
_clone_command_for_github_like(git_config, dest_dir)
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/git_utils.py", line 181, in _clone_command_for_github_like
_clone_command_for_ssh(git_config, dest_dir)
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/git_utils.py", line 192, in _clone_command_for_ssh
_run_clone_command(git_config["repo"], dest_dir)
File "/home/timor/.cache/pypoetry/virtualenvs/edgify-sagemaker--PoVfFez-py3.7/lib/python3.7/site-packages/sagemaker/git_utils.py", line 287, in _run_clone_command
subprocess.check_call(["git", "clone", repo_url, dest_dir], env=my_env)
File "/usr/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', '[email protected]:/.git', '/tmp/tmp9kr9rdwo']' returned non-zero exit status 128.

[mgabilo]

mgabilo@5a70c5d

the ssh url starting with git@ doesn't work for me only ssh://git@

for bug in this thread commenting out the line seemed to do the trick:
my_env["GIT_SSH"] = sshnoprompt.name

[stkbailey]

I had to remove this code block entirely to get it to work, as shown in #3645