Skip to content

infra: support Python 3.7 #1455

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 5, 2020
Merged

infra: support Python 3.7 #1455

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
43870e5
infra: support Python 3.7
May 4, 2020
2f887f1
release build integ tests run against py36
May 4, 2020
2214f75
remove py36 in localmode buildspec ci-script
May 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ SageMaker Python SDK is tested on:

- Python 2.7
- Python 3.6
- Python 3.7

AWS Permissions
~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -117,8 +118,8 @@ You can install the libraries needed to run the tests by running :code:`pip inst


We run unit tests with tox, which is a program that lets you run unit tests for multiple Python versions, and also make sure the
code fits our style guidelines. We run tox with Python 2.7 and 3.6, so to run unit tests
with the same configuration we do, you'll need to have interpreters for Python 2.7 and Python 3.6 installed.
code fits our style guidelines. We run tox with Python 2.7, 3.6 and 3.7, so to run unit tests
with the same configuration we do, you'll need to have interpreters for Python 2.7, Python 3.6 and Python 3.7 installed.

To run the unit tests with tox, run:

Expand Down
4 changes: 2 additions & 2 deletions buildspec-localmodetests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ phases:

# local mode tests
- start_time=`date +%s`
- execute-command-if-has-matching-changes "tox -e py27,py36 -- tests/integ -m local_mode --durations 50" "tests/integ" "tests/data" "tests/conftest.py" "tests/__init__.py" "src/*.py" "setup.py" "setup.cfg" "buildspec-localmodetests.yml"
- ./ci-scripts/displaytime.sh 'py27,py36 local mode' $start_time
- execute-command-if-has-matching-changes "tox -e py27,py37 -- tests/integ -m local_mode --durations 50" "tests/integ" "tests/data" "tests/conftest.py" "tests/__init__.py" "src/*.py" "setup.py" "setup.cfg" "buildspec-localmodetests.yml"
- ./ci-scripts/displaytime.sh 'py27,py36,py37 local mode' $start_time
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should py36 be removed here to match the change in l. 14?

2 changes: 1 addition & 1 deletion buildspec-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ phases:
# run unit tests
- AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_SESSION_TOKEN=
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI= AWS_DEFAULT_REGION=
tox -e py27,py36 -- tests/unit
tox -e py27,py36,py37 -- tests/unit

# run a subset of the integration tests
- IGNORE_COVERAGE=- tox -e py36 -- tests/integ -m canary_quick -n 64 --boxed --reruns 2
Expand Down
4 changes: 2 additions & 2 deletions buildspec-unittests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ phases:
- start_time=`date +%s`
- AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_SESSION_TOKEN=
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI= AWS_DEFAULT_REGION=
tox -e py36,py27 --parallel all -- tests/unit
- ./ci-scripts/displaytime.sh 'py36,py27 unit' $start_time
tox -e py27,py36,py37 --parallel all -- tests/unit
- ./ci-scripts/displaytime.sh 'py27,py36,py37 unit' $start_time
6 changes: 3 additions & 3 deletions buildspec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ phases:

# run integration tests
- start_time=`date +%s`
- execute-command-if-has-matching-changes "python3 -u ci-scripts/queue_build.py" "tests/integ" "tests/scripts" "tests/data" "tests/conftest.py" "tests/__init__.py" "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"
- execute-command-if-has-matching-changes "python3.7 -u ci-scripts/queue_build.py" "tests/integ" "tests/scripts" "tests/data" "tests/conftest.py" "tests/__init__.py" "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"
- ./ci-scripts/displaytime.sh 'build queue' $start_time

- start_time=`date +%s`
- |
execute-command-if-has-matching-changes "env -u AWS_DEFAULT_REGION tox -e py36 -- tests/integ -m \"not local_mode\" -n 512 --reruns 3 --reruns-delay 5 --durations 50 --boto-config '{\"region_name\": \"us-east-2\"}'" "tests/integ" "tests/scripts" "tests/data" "tests/conftest.py" "tests/__init__.py" "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"
- ./ci-scripts/displaytime.sh 'py36 tests/integ' $start_time
execute-command-if-has-matching-changes "env -u AWS_DEFAULT_REGION tox -e py37 -- tests/integ -m \"not local_mode\" -n 512 --reruns 3 --reruns-delay 5 --durations 50 --boto-config '{\"region_name\": \"us-east-2\"}'" "tests/integ" "tests/scripts" "tests/data" "tests/conftest.py" "tests/__init__.py" "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"
- ./ci-scripts/displaytime.sh 'py37 tests/integ' $start_time

post_build:
finally:
Expand Down
1 change: 1 addition & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ def read_version():
"Programming Language :: Python",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
],
install_requires=required_packages,
extras_require=extras,
Expand Down
11 changes: 7 additions & 4 deletions src/sagemaker/tensorflow/estimator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,13 @@ def __init__(self, estimator, logdir=None):
@staticmethod
def _cmd_exists(cmd):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

out of scope, but thought I'd mention it for future reference - it looks like there are better ways to implement this function 😂: https://stackoverflow.com/a/34177358. Maybe we'll look into that Python 3.3+ version when we drop Python 2.7 support.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like shutil does not have which module in python 2.7.

So we can adopt this change when we drop python 2.7 support.

"""Placeholder docstring"""
return any(
os.access(os.path.join(path, cmd), os.X_OK)
for path in os.environ["PATH"].split(os.pathsep)
)
for path in os.environ["PATH"].split(os.pathsep):
try:
if os.access(os.path.join(path, cmd), os.X_OK):
return True
except StopIteration:
return False
return False

@staticmethod
def _sync_directories(from_directory, to_directory):
Expand Down
2 changes: 1 addition & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# and then run "tox" from this directory.

[tox]
envlist = black-format,flake8,pylint,twine,sphinx,doc8,py27,py36
envlist = black-format,flake8,pylint,twine,sphinx,doc8,py27,py36,py37

skip_missing_interpreters = False

Expand Down