Skip to content

Add codecov #4448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 31, 2019
Merged

Add codecov #4448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c1b1d07
Add AWS CodeBuild spec to run codecov
yumibagge May 23, 2019
0ef5e78
Add CPROVER_WITH_PROFILING to make build
yumibagge May 24, 2019
69dbdef
Add codecov badge
yumibagge May 30, 2019
0ea62e3
Add AWS CodeBuild spec to run codecov w cmake
yumibagge May 30, 2019
a780511
Add goto-harness dependency and unify format
yumibagge May 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,10 @@ if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang" OR
COMMAND ${CODE_COVERAGE_LCOV} ${LCOV_FLAGS} --remove ${CODE_COVERAGE_INFO_FILE} '/usr/*' --output-file ${CODE_COVERAGE_INFO_FILE}
COMMAND ${CODE_COVERAGE_GENHTML} ${CODE_COVERAGE_INFO_FILE} --output-directory ${CODE_COVERAGE_OUTPUT_DIR}
DEPENDS
java-models-library java-unit unit
java-models-library
"$<TARGET_FILE:java-unit>"
"$<TARGET_FILE:unit>"
"$<TARGET_FILE:goto-harness>"
"$<TARGET_FILE:cbmc>"
"$<TARGET_FILE:driver>"
"$<TARGET_FILE:goto-analyzer>"
Expand Down
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
![Build Status][codebuild_img]
![Build Status][codebuild_windows_img]
[![Build Status][coverity_img]][coverity]
[![Build Status][codecov_img]][codecov]

[CProver Wiki](http://www.cprover.org/wiki)

Expand Down Expand Up @@ -59,3 +60,5 @@ License
[codebuild_windows_img]: https://codebuild.us-east-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiTFQ4Q0lCSEc1Rk5NcmlzaFZDdU44Vk8zY0c1VCtIVWMwWnJMRitmVFI5bE94Q3dhekVPMWRobFU2Q0xTTlpDSWZUQ3J1eksrWW1rSll1OExXdll2bExZPSIsIml2UGFyYW1ldGVyU3BlYyI6InpqcloyaEdxbjBiQUtvNysiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=develop
[coverity]: https://scan.coverity.com/projects/diffblue-cbmc
[coverity_img]: https://scan.coverity.com/projects/13552/badge.svg
[codecov]: https://codecov.io/gh/diffblue/cbmc
[codecov_img]: https://codecov.io/gh/diffblue/cbmc/branch/develop/graphs/badge.svg
37 changes: 37 additions & 0 deletions buildspec-linux-cmake-gcc-cov.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
version: 0.2

env:
variables:
# CodeBuild console doesn't display color codes correctly
TESTPL_COLOR_OUTPUT: 0

phases:
install:
runtime-versions:
java: openjdk8
commands:
- sed -i 's#/archive.ubuntu.com#/us-east-1.ec2.archive.ubuntu.com#g' /etc/apt/sources.list
- add-apt-repository ppa:ubuntu-toolchain-r/test
- apt-get update -y
- apt-get install -y flex bison make git libwww-perl patch ccache libc6-dev-i386 jq lcov cmake curl gdb python-pip
- pip install codecov
build:
commands:
- echo Build started on `date`
- git submodule update --init --recursive
- cmake -H. -Bbuild '-Denable_coverage=1' '-Dparallel_tests=2' '-DCMAKE_CXX_COMPILER=/usr/bin/g++'
- cmake --build build --target coverage -- -j2
post_build:
commands:
- lcov --capture --directory build --output-file ./lcov.info
- VCS_PULL_REQUEST=$(echo $CODEBUILD_SOURCE_VERSION | sed 's/pr\///g')
- COV_SCRIPT=/root/.cache/codecov.sh
- if [ ! -f "$COV_SCRIPT" ]; then curl -s https://codecov.io/bash > "$COV_SCRIPT"; fi
- echo "$CODEBUILD_INITIATOR" | grep GitHub && bash "$COV_SCRIPT" -t "$CODECOV_TOKEN" || true
- echo Build completed on `date`
cache:
paths:
- '/var/cache/apt/**/*'
- '/var/lib/apt/lists/**/*'
- '/root/.ccache/**/*'
- '/root/.cache/codecov.sh'
45 changes: 45 additions & 0 deletions buildspec-linux-make-gcc-cov.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
version: 0.2

env:
variables:
# CodeBuild console doesn't display color codes correctly
TESTPL_COLOR_OUTPUT: 0

phases:
install:
runtime-versions:
java: openjdk8
commands:
- sed -i 's#/archive.ubuntu.com#/us-east-1.ec2.archive.ubuntu.com#g' /etc/apt/sources.list
- apt-get update -y
- apt-get install -y flex bison make git libwww-perl patch ccache libc6-dev-i386 jq gdb lcov curl python-pip
- pip install codecov
build:
commands:
- echo Build started on `date`
- make -C src minisat2-download
- make -C jbmc/src setup-submodules
- make -C src CXX="ccache g++" -j2 CPROVER_WITH_PROFILING=1
- make -C unit CXX="ccache g++" -j2 CPROVER_WITH_PROFILING=1
- make -C jbmc/src CXX="ccache g++" -j2 CPROVER_WITH_PROFILING=1
- make -C jbmc/unit CXX="ccache g++" -j2 CPROVER_WITH_PROFILING=1
post_build:
commands:
- make -C unit test
- make -C regression test CPROVER_WITH_PROFILING=1
- make -C regression/cbmc test-paths-lifo
- env PATH=$PATH:`pwd`/src/solvers make -C regression/cbmc test-cprover-smt2
- make -C jbmc/unit test
- make -C jbmc/regression test
- lcov --capture --directory . --output-file ./lcov.info
- VCS_PULL_REQUEST=$(echo $CODEBUILD_SOURCE_VERSION | sed 's/pr\///g')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is VCS_PULL_REQUEST actually used anywhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not too sure. As I referred in another your question. I took this 4 lines of scripts from codecov/codecov-python#140.

- COV_SCRIPT=/root/.cache/codecov.sh
- if [ ! -f "$COV_SCRIPT" ]; then curl -s https://codecov.io/bash > "$COV_SCRIPT"; fi
- echo "$CODEBUILD_INITIATOR" | grep GitHub && bash "$COV_SCRIPT" -t "$CODECOV_TOKEN" || true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't fully understand what this line is doing (I'm afraid I don't know what the contents of $CODEBUILD_INITIATOR is off the top of my head) - but I'm slightly worried by the || true - which will basically mask any errors that occur here?

Copy link
Contributor Author

@yumibagge yumibagge May 30, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, essentially I blindly used script example here
codecov/codecov-python#140
According to https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html, $CODEBUILD_INITIATOR is "the entity that started the build". In our case, (also from log), I believe this is webhook?
[Container] 2019/05/30 09:35:34 Running command echo "$CODEBUILD_INITIATOR" | grep GitHub && bash "$COV_SCRIPT" -t "$CODECOV_TOKEN" || true
GitHub-Hookshot/0a2cefb

Do you want me to try removing the || true?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm honestly not sure on the || true - I guess lets run it as is and see how reliable it is.

- echo Build completed on `date`
cache:
paths:
- '/var/cache/apt/**/*'
- '/var/lib/apt/lists/**/*'
- '/root/.ccache/**/*'
- '/root/.cache/codecov.sh'
5 changes: 5 additions & 0 deletions src/config.inc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ else
CXXFLAGS += -Wall -pedantic -Werror -Wno-deprecated-declarations -Wswitch-enum
endif

ifeq ($(CPROVER_WITH_PROFILING),1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you mentioned that somehow CMake unit tests fail if coverage is enabled, but I think it would still be preferable if we kept Make and CMake builds in sync, and at least adding these options to CMake would be good - even if you raise an issue in the GitHub issue tracker to cover whatever failures you see in CMake unit tests rather than also fixing them in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. The thing is this invariant violation happens only codebuild, and I've tried to reproduce it on local codebuild docker, it did not reproduce. I have a gut feeling though initially, cbmc-codecov environment was openjdk-8, not standard-2.0. Let me give me another go with cmake-coverage and if it's reproducible, I will raise an issue in the GitHub.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrisr-diffblue , OK, so perhaps the invariant violation I observed was because of AWS image. I no longer observe it. I still have a problem codecov does not find existing coverage files from the VM. I will push just one more time with extra LCOV command it should be able to report. In that case, do you want another buildspec file buildspec-linux-cmake-gcc-cov.yml and CMakeLists.txt to be part of this PR?

CXXFLAGS += -fprofile-arcs -ftest-coverage
LINKFLAGS += -lgcov -fprofile-arcs
endif

# Select optimisation or debug info
#CXXFLAGS += -O2 -DNDEBUG
#CXXFLAGS += -O0 -g
Expand Down