Merge branch 'master' into pytorch

Author: nadiaya

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,14 @@
+*Issue #, if available:*
+
+*Description of changes:*
+
+## Merge Checklist
+
+_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request._
+
+- [ ] I have read the [CONTRIBUTING](https://github.com/aws/sagemaker-python-sdk/blob/master/CONTRIBUTING.md) doc
+- [ ] I have added tests that prove my fix is effective or that my feature works (if appropriate)
+- [ ] I have updated the [changelog](https://github.com/aws/sagemaker-python-sdk/blob/master/CHANGELOG.rst) with a description of my changes (if appropriate)
+- [ ] I have updated any necessary [documentation](https://github.com/aws/sagemaker-python-sdk/blob/master/README.rst) (if appropriate)
+
+By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

.gitignore

@@ -18,6 +18,8 @@ scratch*.py
 examples/tensorflow/distributed_mnist/data
 *.iml
 doc/_build
+doc/_static
+doc/_templates
 **/.DS_Store
 venv/
 *~

.rtd-environment.yml

@@ -0,0 +1,8 @@
+# Conda environment file used by our custom ReadTheDocs environment.
+
+name: sagemaker
+
+dependencies:
+    - python>=3
+    - numpy
+    - scipy

CHANGELOG.rst

@@ -2,6 +2,10 @@
 CHANGELOG
 =========
 
+1.2.3
+=========
+* bug-fix: Fix local mode not using the right s3 bucket
+
 1.2.2
 =====
 

CODE_OF_CONDUCT.md

@@ -0,0 +1,4 @@
+## Code of Conduct
+This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 
+For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 
+[email protected] with any additional questions or comments.

CONTRIBUTING.md

@@ -0,0 +1,63 @@
+# Contributing Guidelines
+
+Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 
+documentation, we greatly value feedback and contributions from our community.
+
+Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 
+information to effectively respond to your bug report or contribution.
+
+
+## Reporting Bugs/Feature Requests
+
+We welcome you to use the GitHub issue tracker to report bugs or suggest features.
+
+When filing an issue, please check [existing open](https://github.com/aws/sagemaker-python-sdk/issues), or [recently closed](https://github.com/aws/sagemaker-python-sdk/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 
+reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
+
+* A reproducible test case or series of steps
+* The version of our code being used
+* Any modifications you've made relevant to the bug
+* A description of your environment or deployment
+
+
+## Contributing via Pull Requests
+Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
+
+1. You are working against the latest source on the *master* branch.
+2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
+3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
+
+To send us a pull request, please:
+
+1. Fork the repository.
+2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
+3. Include unit tests when you contribute new features or make bug fixes, as they help to a) prove that your code works correctly, and b) guard against future breaking changes to lower the maintenance cost.
+4. Ensure local tests pass.
+5. Update the [changelog](https://github.com/aws/sagemaker-python-sdk/blob/master/CHANGELOG.rst) with a description of your changes for any code contribution.
+6. Commit to your fork using [clear and imperative commit messages](https://chris.beams.io/posts/git-commit/).
+7. Send us a pull request, answering any default questions in the pull request interface.
+8. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
+
+GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 
+[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
+
+
+## Finding contributions to work on
+Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws/sagemaker-python-sdk/labels/help%20wanted) issues is a great place to start. 
+
+
+## Code of Conduct
+This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 
+For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 
+[email protected] with any additional questions or comments.
+
+
+## Security issue notifications
+If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
+
+
+## Licensing
+
+See the [LICENSE](https://github.com/aws/sagemaker-python-sdk/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
+
+We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes.

README.rst

@@ -47,7 +47,7 @@ You can install from source by cloning this repository and issuing a pip install
 
     git clone https://github.com/aws/sagemaker-python-sdk.git
     python setup.py sdist
-    pip install dist/sagemaker-1.2.2.tar.gz
+    pip install dist/sagemaker-1.2.3.tar.gz
 
 Supported Python versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -56,7 +56,8 @@ SageMaker Python SDK is tested on: \* Python 2.7 \* Python 3.5
 
 Licensing
 ~~~~~~~~~
-SageMaker Python SDK is licensed under the Apache 2.0 License. It is copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. The license is available at:  http://aws.amazon.com/apache2.0/
+SageMaker Python SDK is licensed under the Apache 2.0 License. It is copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. The license is available at:  
+http://aws.amazon.com/apache2.0/
 
 Running tests
 ~~~~~~~~~~~~~

doc/conf.py

@@ -15,10 +15,11 @@ def __getattr__(cls, name):
 
 
 MOCK_MODULES = ['tensorflow', 'tensorflow.core', 'tensorflow.core.framework', 'tensorflow.python',
-                'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis']
+                'tensorflow.python.framework', 'tensorflow_serving', 'tensorflow_serving.apis',
+                'numpy', 'scipy', 'scipy.sparse']
 sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
 
-version = '1.2.2'
+version = '1.2.3'
 project = u'sagemaker'
 
 # Add any Sphinx extension module names here, as strings. They can be extensions

readthedocs.yml

@@ -0,0 +1,9 @@
+# ReadTheDocs environment customization to allow us to use conda to install
+# libraries which have C dependencies for the doc build. See:
+# http://docs.readthedocs.io/en/latest/yaml-config.html
+
+conda:
+    file: .rtd-environment.yml
+
+python:
+   setup_py_install: true

setup.py

@@ -11,7 +11,7 @@ def read(fname):
 
 
 setup(name="sagemaker",
-      version="1.2.pytorch",
+      version="1.3.dev",
       description="Open source library for training and deploying models on Amazon SageMaker.",
       packages=find_packages('src'),
       package_dir={'': 'src'},

src/sagemaker/local/image.py

@@ -87,7 +87,6 @@ def train(self, input_data_config, hyperparameters):
         os.mkdir(os.path.join(self.container_root, 'output'))
 
         data_dir = self._create_tmp_folder()
-        bucket_name = self.sagemaker_session.default_bucket()
         volumes = []
 
         # Set up the channels for the containers. For local data we will
@@ -102,7 +101,8 @@ def train(self, input_data_config, hyperparameters):
             channel_dir = os.path.join(data_dir, channel_name)
             os.mkdir(channel_dir)
 
-            if uri.lower().startswith("s3://"):
+            if parsed_uri.scheme == 's3':
+                bucket_name = parsed_uri.netloc
                 self._download_folder(bucket_name, key, channel_dir)
             else:
                 volumes.append(_Volume(uri, channel=channel_name))

tests/integ/test_tf_cifar.py

@@ -31,7 +31,7 @@ def __call__(self, data):
 
 
 def test_cifar(sagemaker_session, tf_full_version):
-    with timeout(minutes=15):
+    with timeout(minutes=20):
         script_path = os.path.join(DATA_DIR, 'cifar_10', 'source')
 
         dataset_path = os.path.join(DATA_DIR, 'cifar_10', 'data')

tests/unit/test_image.py

@@ -16,7 +16,7 @@
 
 import pytest
 import yaml
-from mock import patch, Mock
+from mock import call, patch, Mock
 
 import sagemaker
 from sagemaker.local.image import _SageMakerContainer
@@ -40,7 +40,7 @@
             'S3DataSource': {
                 'S3DataDistributionType': 'FullyReplicated',
                 'S3DataType': 'S3Prefix',
-                'S3Uri': 's3://foo/bar'
+                'S3Uri': 's3://my-own-bucket/prefix'
             }
         }
     }
@@ -54,12 +54,12 @@ def sagemaker_session():
     boto_mock.client('sts').get_caller_identity.return_value = {'Account': '123'}
     boto_mock.resource('s3').Bucket(BUCKET_NAME).objects.filter.return_value = []
 
-    ims = sagemaker.Session(boto_session=boto_mock, sagemaker_client=Mock())
+    sms = sagemaker.Session(boto_session=boto_mock, sagemaker_client=Mock())
 
-    ims.default_bucket = Mock(name='default_bucket', return_value=BUCKET_NAME)
-    ims.expand_role = Mock(return_value=EXPANDED_ROLE)
+    sms.default_bucket = Mock(name='default_bucket', return_value=BUCKET_NAME)
+    sms.expand_role = Mock(return_value=EXPANDED_ROLE)
 
-    return ims
+    return sms
 
 
 @patch('sagemaker.local.local_session.LocalSession')
@@ -181,16 +181,22 @@ def test_check_output():
 @patch('sagemaker.local.local_session.LocalSession')
 @patch('sagemaker.local.image._execute_and_stream_output')
 @patch('sagemaker.local.image._SageMakerContainer._cleanup')
-def test_train(LocalSession, _execute_and_stream_output, _cleanup, tmpdir, sagemaker_session):
+@patch('sagemaker.local.image._SageMakerContainer._download_folder')
+def test_train(_download_folder, _cleanup, _execute_and_stream_output, LocalSession, tmpdir, sagemaker_session):
 
+    directories = [str(tmpdir.mkdir('container-root')), str(tmpdir.mkdir('data'))]
     with patch('sagemaker.local.image._SageMakerContainer._create_tmp_folder',
-               side_effect=[str(tmpdir.mkdir('container-root')), str(tmpdir.mkdir('data'))]):
+               side_effect=directories):
 
         instance_count = 2
         image = 'my-image'
         sagemaker_container = _SageMakerContainer('local', instance_count, image, sagemaker_session=sagemaker_session)
         sagemaker_container.train(INPUT_DATA_CONFIG, HYPERPARAMETERS)
 
+        channel_dir = os.path.join(directories[1], 'b')
+        download_folder_calls = [call('my-own-bucket', 'prefix', channel_dir)]
+        _download_folder.assert_has_calls(download_folder_calls)
+
         docker_compose_file = os.path.join(sagemaker_container.container_root, 'docker-compose.yaml')
 
         call_args = _execute_and_stream_output.call_args[0][0]
@@ -231,6 +237,36 @@ def test_serve(up, copy, copytree, tmpdir, sagemaker_session):
                 assert config['services'][h]['command'] == 'serve'
 
 
+@patch('os.makedirs')
+def test_download_folder(makedirs):
+    boto_mock = Mock(name='boto_session')
+    boto_mock.client('sts').get_caller_identity.return_value = {'Account': '123'}
+
+    session = sagemaker.Session(boto_session=boto_mock, sagemaker_client=Mock())
+
+    train_data = Mock()
+    validation_data = Mock()
+
+    train_data.bucket_name.return_value = BUCKET_NAME
+    train_data.key = '/prefix/train/train_data.csv'
+    validation_data.bucket_name.return_value = BUCKET_NAME
+    validation_data.key = '/prefix/train/validation_data.csv'
+
+    s3_files = [train_data, validation_data]
+    boto_mock.resource('s3').Bucket(BUCKET_NAME).objects.filter.return_value = s3_files
+
+    obj_mock = Mock()
+    boto_mock.resource('s3').Object.return_value = obj_mock
+
+    sagemaker_container = _SageMakerContainer('local', 2, 'my-image', sagemaker_session=session)
+    sagemaker_container._download_folder(BUCKET_NAME, '/prefix', '/tmp')
+
+    obj_mock.download_file.assert_called()
+    calls = [call(os.path.join('/tmp', 'train/train_data.csv')),
+             call(os.path.join('/tmp', 'train/validation_data.csv'))]
+    obj_mock.download_file.assert_has_calls(calls)
+
+
 def test_ecr_login_non_ecr():
     session_mock = Mock()
     sagemaker.local.image._ecr_login_if_needed(session_mock, 'ubuntu')