Skip to content

change: allow only one integration test run per time #880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Jun 27, 2019
Merged
Changes from 2 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
131ec2a
change: allow only one integration test run per time
mvsusp Jun 25, 2019
1c3c673
Update buildspec.yml
mvsusp Jun 25, 2019
bde21ff
Update buildspec.yml
mvsusp Jun 25, 2019
cb94061
Update buildspec.yml
mvsusp Jun 25, 2019
bc2fa85
Update buildspec.yml
mvsusp Jun 25, 2019
9c7367d
change: allow only one integration test run per time
mvsusp Jun 25, 2019
6b54703
Merge branch 'mvs-cilock' of github.com:mvsusp/sagemaker-python-sdk i…
mvsusp Jun 25, 2019
7f94164
change: allow only one integration test run per time
mvsusp Jun 25, 2019
5da7012
change: allow only one integration test run per time
mvsusp Jun 25, 2019
b420893
change: allow only one integration test run per time
mvsusp Jun 25, 2019
5273782
change: allow only one integration test run per time
mvsusp Jun 25, 2019
266d7be
change: allow only one integration test run per time
mvsusp Jun 25, 2019
0e79d18
change: allow only one integration test run per time
mvsusp Jun 25, 2019
e4d6505
change: allow only one integration test run per time
mvsusp Jun 25, 2019
6c7af65
change: allow only one integration test run per time
mvsusp Jun 25, 2019
1a5fd2a
change: allow only one integration test run per time
mvsusp Jun 25, 2019
ef94d66
change: allow only one integration test run per time
mvsusp Jun 25, 2019
21ee422
change: allow only one integration test run per time
mvsusp Jun 25, 2019
d5b46aa
change: allow only one integration test run per time
mvsusp Jun 25, 2019
29bdd84
change: allow only one integration test run per time
mvsusp Jun 25, 2019
6f3e7d4
change: allow only one integration test run per time
mvsusp Jun 25, 2019
3877f2d
Merge branch 'master' into mvs-cilock
mvsusp Jun 25, 2019
eec3f3a
Merge branch 'master' into mvs-cilock
mvsusp Jun 26, 2019
7916612
Merge branch 'master' into mvs-cilock
mvsusp Jun 26, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,41 @@ phases:

# run notebook test
- |
if has-matching-changes "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"; then
if has-matching-changes "src/*.py" "setup.py" "setup.cfg"; then
echo "running notebook test"
./tests/scripts/run-notebook-test.sh
else
echo "skipping notebook test"
fi

# wait its turn to run integration tests
- ACCOUNT=$(aws sts get-caller-identity --output text | awk '{print $1}')
- S3_BUCKET_DIR=s3://sagemaker-us-west-2-${ACCOUNT}/ci-lock

- |
while true; do
LAST_LOCK=$(aws s3 ls ${S3_BUCKET_DIR} | sort | tail -n 1 | awk '{print $4}')

if test -z "$LAST_LOCK"
then
echo "No locks. Creating lock."
LOCK_FILE=$(date +%s)
touch ${LOCK_FILE}

aws s3 cp ${LOCK_FILE} ${S3_BUCKET_DIR}/${LOCK_FILE}

else
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per a locking system that was implemented in the Shopbop order system. You'll also want to ignore a file that is more than x hours old. To ensure dead processes don't lock forever.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The process won't lock forever because the code build has a time limit as well. Let's include this as a future improvement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is that a CodeBuild process dies or times out and leaves a file in S3.
In that situation, manual intervention is required to allow any future builds to run.

echo "Waiting on lock."
sleep 30
fi
done

# run integration tests
- |
if has-matching-changes "tests/" "src/*.py" "setup.py" "setup.cfg" "buildspec.yml"; then
IGNORE_COVERAGE=- tox -e py36,py27 -- tests/integ -n 24 --boxed --reruns 2
else
echo "skipping integration tests"
fi
finally:
- aws s3 rm --recursive ${S3_BUCKET_DIR}