feature: sts endpoint support

[jmgray24]

Issue #, if available: N/A

Description of changes:

Added the ability to pass an optional STS endpoint URL. This optional STS url allows the override of the global endpoint for regional ones.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

• I have read the CONTRIBUTING doc
• I used the commit message format described in CONTRIBUTING
• I have added tests that prove my fix is effective or that my feature works (if appropriate)
• I have updated any necessary documentation (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jmgray24]

This appears to be a issue with the build outside of the scope of the changes in this PR as it is also seen in #786.

Thus ready for review.

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[imujjwal96]

The issue has been fixed in the PR #776
you may run the following commands to pull the latest commits:

git remote add upstream https://github.com/aws/sagemaker-python-sdk.git
git pull --rebase upstream master
git push origin master

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jmgray24]

Failure of test here

_________________________ test_horovod_local_mode[2-2] _________________________
[gw9] linux -- Python 3.6.7 /codebuild/output/src402338386/src/github.com/aws/sagemaker-python-sdk/.tox/py36/bin/python
sagemaker_local_session = <sagemaker.local.local_session.LocalSession object at 0x7fda65dae748>
instances = 2, processes = 2
tmpdir = local('/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0')

@pytest.mark.local_mode
@pytest.mark.parametrize('instances, processes', [
    [1, 2],
    (2, 1),
    (2, 2)])
def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):
    output_path = 'file://%s' % tmpdir
    job_name = sagemaker.utils.unique_name_from_base('tf-horovod')
    estimator = TensorFlow(entry_point=os.path.join(horovod_dir, 'test_hvd_basic.py'),
                           role='SageMakerRole',
                           train_instance_count=2,
                           train_instance_type='local',
                           sagemaker_session=sagemaker_local_session,
                           py_version=integ.PYTHON_VERSION,
                           script_mode=True,
                           output_path=output_path,
                           framework_version='1.12',
                           distributions={'mpi': {'enabled': True,
                                                  'processes_per_host': processes}})

    with timeout.timeout(minutes=integ.TRAINING_DEFAULT_TIMEOUT_MINUTES):
        estimator.fit(job_name=job_name)

        tmp = str(tmpdir)
        extract_files(output_path.replace('file://', ''), tmp)

        size = instances * processes

        for rank in range(size):

          assert read_json('rank-%s' % rank, tmp)['rank'] == rank

tests/integ/test_horovod.py:93:

---

file = 'rank-0'
tmp = '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0'

def read_json(file, tmp):

  with open(os.path.join(tmp, file)) as f:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-2/popen-gw9/test_horovod_local_mode_2_2_0/rank-0'

tests/integ/test_horovod.py:102: FileNotFoundError
----------------------------- Captured stdout call -----------------------------

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jesterhazy]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jesterhazy]

AWS CodeBuild CI Report

• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jmgray24]

Closing, as I need to change this entirely. Because of the way deploy and other methods call get_default_bucket, This will be a poor design pattern. Instead, I will need to pass the STS endpoint during SageMaker session initialization and then check STS endpoint URL in the functions.