Enable new SMT backend test suite to be run against cvc5 locally and on CI #6777
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7 tasks
regression/cbmc-incr-smt2/nondeterministic-int-assert/incremental_solver_called.desc
Outdated
Show resolved
Hide resolved
|
This will need the CVC5 version number being installed in CI to be bumped before the tests are going to pass. The fact that the make build jobs are currently passing suggests that the CVC5 tests are not being run/checked properly on these jobs. |
NlightNFotis
force-pushed
the
run_tests_against_cvc5
branch
3 times, most recently
from
d708c65 to
c0c15b1
Compare
TGWDB
reviewed
Apr 4, 2022
regression/cbmc-incr-smt2/Makefile
Outdated
| @../test.pl -e -p -c "../../../src/cbmc/cbmc --incremental-smt2-solver 'z3 --smt2 -in' --validate-goto-model --validate-ssa-equation --slice-formula" $(exclude_broken_windows_tests) | ||
|
|
||
| test.cvc5: | ||
| @../test.pl -e -p -c "../../../src/cbmc/cbmc --incremental-smt2-solver 'cvc5 --lang smt --incremental' --validate-goto-model --validate-ssa-equation --slice-formula" $(exclude_broken_windows_tests) |
There was a problem hiding this comment.
Yep, getting on to that.
regression/cbmc-incr-smt2/bitvector-bitwise-operators/shift_right.desc
Outdated
Show resolved
Hide resolved
This allows us to test against more than one solvers in the new backend (currently tested - cvc5, z3). This ensures that we don't invariably issue output that is solver specific, tying the new backend to a single solver by accident.
…ith multiple solvers. Previously the command that was enabling the new SMT backend (and providing the script for invoking the appropriate SMT solver) was passed as part of the options passed in `test.desc`. Since this was removed, and passed as part of the test configuration in the `CMakeLists.txt`, we need to do the same here.
NlightNFotis
force-pushed
the
run_tests_against_cvc5
branch
from
c0c15b1 to
fbd7662
Compare
thomasspriggs
approved these changes
Apr 4, 2022
TGWDB
approved these changes
Apr 4, 2022
tautschnig
added a commit
to tautschnig/cbmc
that referenced
this pull request
Apr 8, 2022
As of diffblue#6777, the codecov job kept failing for CVC5 wasn't available in this job. The same would have been true of release jobs, but we haven't had one since diffblue#6777 was merged.
3 tasks
tautschnig
added a commit
to tautschnig/cbmc
that referenced
this pull request
Apr 8, 2022
As of diffblue#6777, the codecov job kept failing for CVC5 wasn't available in this job. The same would have been true of release jobs, but we haven't had one since diffblue#6777 was merged.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the evolution of #6745, given that the bug we were working around has now been fixed by CVC5 developers (here -> cvc5/cvc5#8489).
This allows for a cleaner changeset here, where in we only make changes to tests
and test configuration files to allow to run with multiple solvers (CVC5 and z3).