Skip to content

chore: Fail build if code coverage is too low #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 26, 2021

Conversation

farleyb-amazon
Copy link
Contributor

@farleyb-amazon farleyb-amazon commented Mar 22, 2021

Description of changes:
Remove code coverage from base python command, add a dedicated code coverage environment, as well as code build specs to run it.

If we have all test environments running code coverage and configure it to fail on low coverage, we run into issues with things like integration tests, which intentionally are not exhaustive in their coverage. Plus running coverage on every test isn't that useful without a mechanism to do something about it (hoping a human happens to look at the output doesn't count).

So, make code coverage a dedicated target, have it target unit tests, and set a high threshold.

See also: aws/aws-dynamodb-encryption-python#153

Testing:

Building gives me:

Required test coverage of 95.0% reached. Total coverage: 98.46%

If I bump the threshold to 99, the build fails with:

FAIL Required test coverage of 99.0% not reached. Total coverage: 98.46%

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Check any applicable:

  • Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.

robin-aws
robin-aws previously approved these changes Mar 22, 2021
@@ -9,6 +9,7 @@ branch = True

[coverage:report]
show_missing = True
fail_under = 95
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record I think 95% is almost a bit too high - I like to aim for 90% so we don't hit diminishing returns, given code coverage is just an imprecise measurement. But given we're already this high I'm willing to see whether we can keep it up without feeling like we're writing a lot of low-value tests.

@farleyb-amazon farleyb-amazon merged commit 7bbfb76 into aws:master Mar 26, 2021
@farleyb-amazon farleyb-amazon deleted the code-coverage branch March 26, 2021 17:12
robin-aws pushed a commit to robin-aws/aws-encryption-sdk-python that referenced this pull request Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants