[Bug] Environment Variables through Estimators in Local Mode Does Not Work

[ssmall41]

Describe the bug
Estimators accept an argument for setting environment variables in training jobs. However providing values here does not actually set them in the container when using local mode. A workaround is to pass the values as hyperparameters and then use them appropriately in the container code or training script.

To reproduce
For example, with the PyTorch estimator:

from sagemaker.pytorch import PyTorch
estimator = PyTorch(instance_type="local", environment={"MYVAR": "got_it"}, ...)

Calls to os.environ["MYVAR"] do not see "MYVAR" defined. If instead I use instance_type="ml.g4dn.xlarge", then "MYVAR" is defined in the container, as expected.

Expected behavior
"MYVAR" should be defined in the container.

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.72.3
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
• Framework version: 1.9.0
• Python version: py38
• CPU or GPU: GPU
• Custom Docker image (Y/N): No

I can also produce this with Tensorflow images as well, so I suspect it's not dependent on the framework.

[aamster]

There should at least be a warning that the environment variables are being ignored because of local mode

[mufaddal-rohawala]

Thanks for you feedback @ssmall41 and @aamster, and agree there should have been warning.

With #3015 this issue should now be resolved.