local mode permission denied when file is written from running job to mounted volume

[jmahlik]

Describe the bug
Permission denied in local mode when files are created by the running job. This occurs on the ProcessingOutput volumes when attempting to clean up. The job doesn't exit cleanly.

Fix is here:

#3017

To reproduce
Write a file to a mounted docker volume as a non-root user using local mode.

Expected behavior
Try except the error.

Screenshots or logs

---> 76     shutil.rmtree(source)
     77     return final_uri
...
PermissionError: [Errno 13] Permission denied: 'myfile2.txt'

System information
A description of your system. Please provide:

• SageMaker Python SDK version: latest
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): local mode
• Framework version: n/a
• Python version: python 3.9
• CPU or GPU: cpu
• Custom Docker image (Y/N): Y

Additional context
Add any other context about the problem here.

~local mode

[david-waterworth]

I'm still experiancing with with sagemaker==2.224.2

INFO:sagemaker.local.entities:Pipeline step 'XXX' FAILED. Failure message is: PermissionError: [Errno 13] Permission denied: '/tmp/tmpq_a0b8qh/output/...'

I added some instrumentation and the problem isn't deleting the file, the permission denied comes from the earlier line when the file is uploaded to s3

sagemaker-python-sdk/src/sagemaker/local/utils.py

Line 84 in 0e3769e

sagemaker_session.upload_data(source, bucket, path)