Skip to content

update: Update buildspec for TF 1.15.0 #265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 9, 2020
Merged

update: Update buildspec for TF 1.15.0 #265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b9bec16
Change path for entrypoint script
saimidu Dec 30, 2019
98dedef
Change path for deep learning container script
saimidu Dec 30, 2019
8b760b3
Change build context to folder containing dockerfiles
saimidu Jan 3, 2020
da1a091
Update buildspec
saimidu Jan 3, 2020
52951af
Correct import for dlc test
saimidu Jan 4, 2020
bee22bf
Add init and comply with flake8
saimidu Jan 7, 2020
dd640d8
Add init inside build artifacts dir
saimidu Jan 7, 2020
b7bd984
Update buildspec-release
saimidu Jan 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions buildspec-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 0.2

env:
variables:
FRAMEWORK_VERSION: '1.13.1'
FRAMEWORK_VERSION: '1.15.0'
GPU_INSTANCE_TYPE: 'ml.p2.xlarge'
SETUP_FILE: 'setup_cmds.sh'
SETUP_CMDS: '#!/bin/bash\npip install --upgrade pip\npip install -U -e .\npip install -U -e .[test]'
Expand Down Expand Up @@ -60,21 +60,21 @@ phases:
echo '[{
"repository": "sagemaker-tensorflow-scriptmode",
"tags": [{
"source": "1.13.1-cpu-py2",
"dest": ["1.13.1-cpu-py2", "1.13-cpu-py2", "1.13.1-cpu-py2-'${CODEBUILD_BUILD_ID#*:}'"]
"source": "1.15.0-cpu-py2",
"dest": ["1.15.0-cpu-py2", "1.15-cpu-py2", "1.15.0-cpu-py2-'${CODEBUILD_BUILD_ID#*:}'"]
},{
"source": "1.13.1-cpu-py3",
"dest": ["1.13.1-cpu-py3", "1.13-cpu-py3", "1.13.1-cpu-py3-'${CODEBUILD_BUILD_ID#*:}'"]
"source": "1.15.0-cpu-py3",
"dest": ["1.15.0-cpu-py3", "1.15-cpu-py3", "1.15.0-cpu-py3-'${CODEBUILD_BUILD_ID#*:}'"]
},{
"source": "1.13.1-gpu-py2",
"dest": ["1.13.1-gpu-py2", "1.13-gpu-py2", "1.13.1-gpu-py2-'${CODEBUILD_BUILD_ID#*:}'"]
"source": "1.15.0-gpu-py2",
"dest": ["1.15.0-gpu-py2", "1.15-gpu-py2", "1.15.0-gpu-py2-'${CODEBUILD_BUILD_ID#*:}'"]
},{
"source": "1.13.1-gpu-py3",
"dest": ["1.13.1-gpu-py3", "1.13-gpu-py3", "1.13.1-gpu-py3-'${CODEBUILD_BUILD_ID#*:}'"]
"source": "1.15.0-gpu-py3",
"dest": ["1.15.0-gpu-py3", "1.15-gpu-py3", "1.15.0-gpu-py3-'${CODEBUILD_BUILD_ID#*:}'"]
}],
"test": [
"IGNORE_COVERAGE=- tox -e py36 -- -m deploy_test test/integration/sagemaker -n 4 --region {region} --account-id {aws-id} --instance-type {cpu-instance-type} --docker-base-name sagemaker-tensorflow-scriptmode --framework-version 1.13.1 --processor cpu --py-version 2,3",
"IGNORE_COVERAGE=- tox -e py36 -- -m deploy_test test/integration/sagemaker -n 4 --region {region} --account-id {aws-id} --docker-base-name sagemaker-tensorflow-scriptmode --framework-version 1.13.1 --processor gpu --py-version 2,3"
"IGNORE_COVERAGE=- tox -e py36 -- -m deploy_test test/integration/sagemaker -n 4 --region {region} --account-id {aws-id} --instance-type {cpu-instance-type} --docker-base-name sagemaker-tensorflow-scriptmode --framework-version 1.15.0 --processor cpu --py-version 2,3",
"IGNORE_COVERAGE=- tox -e py36 -- -m deploy_test test/integration/sagemaker -n 4 --region {region} --account-id {aws-id} --docker-base-name sagemaker-tensorflow-scriptmode --framework-version 1.15.0 --processor gpu --py-version 2,3"
]
}]' > deployments.json

Expand Down
79 changes: 48 additions & 31 deletions buildspec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,7 @@ version: 0.2

env:
variables:
FRAMEWORK_VERSION: '1.13.1'
CPU_FRAMEWORK_BINARY: 'https://s3-us-west-2.amazonaws.com/tensorflow-aws/1.13/AmazonLinux/cpu/latest-patch-latest-patch/tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl'
CPU_PY_VERSION: '3'
GPU_FRAMEWORK_BINARY: 'https://s3-us-west-2.amazonaws.com/tensorflow-aws/1.13/AmazonLinux/gpu/latest-patch-latest-patch/tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl'
GPU_PY_VERSION: '3'
FRAMEWORK_VERSION: '1.15.0'
ECR_REPO: 'sagemaker-test'
GITHUB_REPO: 'sagemaker-tensorflow-container'
SETUP_FILE: 'setup_cmds.sh'
Expand Down Expand Up @@ -34,42 +30,56 @@ phases:
- tox -e py36,py27 test/unit

# Create pip archive
- build_dir="docker/$FRAMEWORK_VERSION"
- root_dir=$(pwd)
- build_id="$(echo $CODEBUILD_BUILD_ID | sed -e 's/:/-/g')"
- python3 setup.py sdist
- tar_name=$(ls dist)
- cp dist/$tar_name $build_dir

# build cpu image
- cpu_dockerfile="Dockerfile.cpu"
# Find build artifacts
- build_artifacts=$root_dir/docker/artifacts

# Download framework binary
- cpu_fw_binary=$(basename $CPU_FRAMEWORK_BINARY)
- wget -O $build_dir/$cpu_fw_binary $CPU_FRAMEWORK_BINARY

- CPU_TAG="$FRAMEWORK_VERSION-cpu-py$CPU_PY_VERSION-$build_id"
# build py2 images

# prepare build context
- build_dir="$root_dir/docker/$FRAMEWORK_VERSION/py2"
- cp $root_dir/dist/$tar_name $build_dir
- cp $build_artifacts/* $build_dir/
- cd $build_dir
- docker build -f $cpu_dockerfile --build-arg framework_support_installable=$tar_name --build-arg py_version=$CPU_PY_VERSION --build-arg framework_installable=$cpu_fw_binary -t $PREPROD_IMAGE:$CPU_TAG .
- cd ../../

# build cpu image
- cpu_dockerfile="Dockerfile.cpu"
- CPU_TAG_PY2="$FRAMEWORK_VERSION-cpu-py2-$build_id"
- docker build -f $cpu_dockerfile -t $PREPROD_IMAGE:$CPU_TAG_PY2 .

# build gpu image
- gpu_dockerfile="Dockerfile.gpu"
- GPU_TAG_PY2="$FRAMEWORK_VERSION-gpu-py2-$build_id"
- docker build -f $gpu_dockerfile -t $PREPROD_IMAGE:$GPU_TAG_PY2 .

# Download framework binary
- gpu_fw_binary=$(basename $GPU_FRAMEWORK_BINARY)
- wget -O $build_dir/$gpu_fw_binary $GPU_FRAMEWORK_BINARY

- GPU_TAG="$FRAMEWORK_VERSION-gpu-py$GPU_PY_VERSION-$build_id"
# build py3 images

# prepare build context
- build_dir="$root_dir/docker/$FRAMEWORK_VERSION/py3"
- cp $root_dir/dist/$tar_name $build_dir
- cp $build_artifacts/* $build_dir/
- cd $build_dir
- docker build -f $gpu_dockerfile --build-arg framework_support_installable=$tar_name --build-arg py_version=$GPU_PY_VERSION --build-arg framework_installable=$gpu_fw_binary -t $PREPROD_IMAGE:$GPU_TAG .
- cd ../../

# build cpu image
- cpu_dockerfile="Dockerfile.cpu"
- CPU_TAG_PY3="$FRAMEWORK_VERSION-cpu-py3-$build_id"
- docker build -f $cpu_dockerfile -t $PREPROD_IMAGE:$CPU_TAG_PY3 .

# build gpu image
- gpu_dockerfile="Dockerfile.gpu"
- GPU_TAG_PY3="$FRAMEWORK_VERSION-gpu-py3-$build_id"
- docker build -f $gpu_dockerfile -t $PREPROD_IMAGE:$GPU_TAG_PY3 .

# push images to ecr
- $(aws ecr get-login --registry-ids $ACCOUNT --no-include-email --region $AWS_DEFAULT_REGION)
- docker push $PREPROD_IMAGE:$CPU_TAG
- docker push $PREPROD_IMAGE:$GPU_TAG
- docker push $PREPROD_IMAGE:$CPU_TAG_PY2
- docker push $PREPROD_IMAGE:$GPU_TAG_PY2
- docker push $PREPROD_IMAGE:$CPU_TAG_PY3
- docker push $PREPROD_IMAGE:$GPU_TAG_PY3

# launch remote gpu instance
- instance_type='p2.xlarge'
Expand All @@ -79,7 +89,8 @@ phases:
# run cpu integration tests
- |
if has-matching-changes "test/" "tests/" "src/*.py" "docker/*" "buildspec.yml"; then
pytest test/integration/local --region $AWS_DEFAULT_REGION --docker-base-name $PREPROD_IMAGE --tag $CPU_TAG --framework-version $FRAMEWORK_VERSION --py-version $CPU_PY_VERSION --processor cpu
pytest test/integration/local --region $AWS_DEFAULT_REGION --docker-base-name $PREPROD_IMAGE --tag $CPU_TAG_PY2 --framework-version $FRAMEWORK_VERSION --py-version 2 --processor cpu
pytest test/integration/local --region $AWS_DEFAULT_REGION --docker-base-name $PREPROD_IMAGE --tag $CPU_TAG_PY3 --framework-version $FRAMEWORK_VERSION --py-version 3 --processor cpu
else
echo "skipping cpu integration tests"
fi
Expand All @@ -88,7 +99,9 @@ phases:
- |
if has-matching-changes "test/" "tests/" "src/*.py" "docker/*" "buildspec.yml"; then
printf "$SETUP_CMDS" > $SETUP_FILE
cmd="pytest test/integration/local --region $AWS_DEFAULT_REGION --docker-base-name $PREPROD_IMAGE --tag $GPU_TAG --framework-version $FRAMEWORK_VERSION --py-version $GPU_PY_VERSION --processor gpu"
cmd="pytest test/integration/local --region $AWS_DEFAULT_REGION --docker-base-name $PREPROD_IMAGE --tag $GPU_TAG_PY2 --framework-version $FRAMEWORK_VERSION --py-version 2 --processor gpu"
remote-test --github-repo $GITHUB_REPO --test-cmd "$cmd" --setup-file $SETUP_FILE --pr-number "$PR_NUM"
cmd="pytest test/integration/local --region $AWS_DEFAULT_REGION --docker-base-name $PREPROD_IMAGE --tag $GPU_TAG_PY3 --framework-version $FRAMEWORK_VERSION --py-version 3 --processor gpu"
remote-test --github-repo $GITHUB_REPO --test-cmd "$cmd" --setup-file $SETUP_FILE --pr-number "$PR_NUM"
else
echo "skipping gpu integration tests"
Expand All @@ -97,8 +110,10 @@ phases:
# run sagemaker tests
- |
if has-matching-changes "test/" "tests/" "src/*.py" "docker/*" "buildspec.yml"; then
pytest test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $CPU_TAG --py-version $CPU_PY_VERSION --processor cpu
pytest test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $GPU_TAG --py-version $GPU_PY_VERSION --processor gpu
pytest test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $CPU_TAG_PY2 --py-version 2 --processor cpu
pytest test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $GPU_TAG_PY2 --py-version 2 --processor gpu
pytest test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $CPU_TAG_PY3 --py-version 3 --processor cpu
pytest test/integration/sagemaker -n 8 --region $AWS_DEFAULT_REGION --docker-base-name $ECR_REPO --account-id $ACCOUNT --tag $GPU_TAG_PY3 --py-version 3 --processor gpu
else
echo "skipping sagemaker tests"
fi
Expand All @@ -109,5 +124,7 @@ phases:
- cleanup-key-pairs

# remove ecr image
- aws ecr batch-delete-image --repository-name $ECR_REPO --region $AWS_DEFAULT_REGION --image-ids imageTag=$CPU_TAG
- aws ecr batch-delete-image --repository-name $ECR_REPO --region $AWS_DEFAULT_REGION --image-ids imageTag=$GPU_TAG
- aws ecr batch-delete-image --repository-name $ECR_REPO --region $AWS_DEFAULT_REGION --image-ids imageTag=$CPU_TAG_PY2
- aws ecr batch-delete-image --repository-name $ECR_REPO --region $AWS_DEFAULT_REGION --image-ids imageTag=$GPU_TAG_PY2
- aws ecr batch-delete-image --repository-name $ECR_REPO --region $AWS_DEFAULT_REGION --image-ids imageTag=$CPU_TAG_PY3
- aws ecr batch-delete-image --repository-name $ECR_REPO --region $AWS_DEFAULT_REGION --image-ids imageTag=$GPU_TAG_PY3
Empty file added docker/__init__.py
View file
Open in desktop
Empty file.
Empty file added docker/build_artifacts/__init__.py
View file
Open in desktop
Empty file.
0 ...flow_container/deep_learning_container.py → ...uild_artifacts/deep_learning_container.py
View file
Open in desktop
File renamed without changes.
0 docker/1.15.0/py2/dockerd-entrypoint.py → docker/build_artifacts/dockerd-entrypoint.py
View file
Open in desktop
File renamed without changes.
3 changes: 1 addition & 2 deletions test/unit/test_deep_learning_containers.py → test/unit/test_deep_learning_container.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,10 @@

import unittest

from docker.build_artifacts import deep_learning_container as deep_learning_container_to_test
Copy link
Contributor

@nadiaya nadiaya Jan 9, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this import should go after pytest and request because it's our library

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to do this to comply with flake8 requirements.
I think module names need to be sorted in lexicographical order.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think the reason this is happening is b/c flake8 recognizes "docker" as the docker python sdk, so it thinks this is a 3rd party lib instead of a local one. One possible workaround is to noqa this line for that particular flake8 rule.

import pytest
import requests

from sagemaker_tensorflow_container import deep_learning_container as deep_learning_container_to_test


@pytest.fixture(name='fixture_valid_instance_id')
def fixture_valid_instance_id(requests_mock):
Expand Down