Skip to content

Commit 3f9d01f

Browse files
author
Owen Jones
committed
Address more review comments
1 parent f88a065 commit 3f9d01f

File tree

1 file changed

+50
-14
lines changed

1 file changed

+50
-14
lines changed

doc/architectural/compilation-and-development.md

Lines changed: 50 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,9 @@ relating to compilation so that they can be conveniently edited in one place.
4141
contains commands that are needed in every makefile but which the
4242
developer is not expected to edit. (There is also
4343
[another config.inc](https://github.com/diffblue/cbmc/blob/develop/jbmc/src/config.inc)
44-
which is also included in every makefile in the `jbmc` folder.)
44+
which is also included in every makefile in the `jbmc` folder.)
45+
46+
Note, these files are not involved in the CMake build.
4547

4648

4749
\subsubsection compilation-and-development-subsubsection-macro-debug Macro DEBUG
@@ -59,31 +61,63 @@ The regression tests are contained in `regression/` and `jbmc/regression/`.
5961
Inside these folders there is a directory for each of the tools/modules. Each
6062
of these contains multiple test directories, with names describing
6163
what they test. When there are multiple tests in a test directory then
62-
they should be testing very similar aspects of the program's behaviour. Each
64+
they should all test very similar aspects of the program's behaviour. Each
6365
test directory contains input files and one or more test description files,
6466
which have the ending `.desc`. The test description files describe what command
6567
to run, what output is expected and so on. The test framework is a
6668
Perl script,
67-
[test.pl](https://github.com/diffblue/cbmc/blob/develop/regression/test.pl).
68-
To run all the tests in a directory corresponding to a tool or module:
69-
70-
../test.pl -c PATH_TO_CBMC
69+
[test.pl](https://github.com/diffblue/cbmc/blob/develop/regression/test.pl),
70+
located in `regression/test.pl`.
7171

72-
The `--help` option to `test.pl` gives instructions for use and outlines the
72+
The `--help` option to `test.pl` outlines the
7373
format of the test description files. Most importantly, the first word in a
7474
test description file is its level, which is one of: `CORE` (should be run in
7575
CI, should succeed), `THOROUGH` (takes too long to be run in CI, should
7676
succeed), `FUTURE` (will succeed when a planned feature is added) or
7777
`KNOWNBUG` (will succeed when a bug is fixed).
7878

79-
If you have compiled CBMC using CMake then another way to run the same tests
80-
is through `CTest`. From the build directory, the command `ctest -V -L CORE`
81-
will run all of the tests that are labelled CORE in the desc files and unit
82-
tests. `-V` makes it print out more
83-
useful output and `-L CORE` makes it only run tests that have been tagged
84-
`CORE`. `-R regular_expression` can be used to limit which tests are run, and
85-
`-N` makes it lists which tests it will run without actually running them.
79+
\subsubsection compilation-and-development-subsubsection-running-regression-tests-with-make Running regression tests with `make`
80+
81+
If you have compiled using `make` then you can run the regression tests
82+
using `make test`. Run it from `regression/` to run all the regression tests,
83+
or any of its subfolders to just run the tests in that subfolder.
84+
85+
If you have not compiled using `make` then this won't work, because the
86+
makefile is expecting to find binaries like `cbmc` and `jbmc` in the source
87+
folders.
88+
89+
\subsubsection compilation-and-development-subsubsection-running-regression-tests-with-ctest Running regression tests with `ctest`
90+
91+
If you have compiled using CMake then you can run the regression tests (and
92+
the unit tests) using CTest.
93+
94+
Here are two example commands, to be run from the build directory:
95+
96+
ctest -V -L CORE -R cpp
97+
ctest -V -L CORE -R cpp -E cbmc-cpp
8698

99+
`-V` makes it print out more
100+
useful output. `-L CORE` makes it only run tests that have been tagged
101+
`CORE`. `-R regular_expression` can be used to limit which tests are run to
102+
those which match the given regular expression, and `-E regex` excludes tests
103+
to those which match the given regular expression.
104+
So the first command will run all the CORE tests in `regression/cbmc/cpp` and
105+
`regression/cbmc/cbmc-cpp`, and the second will run all the CORE tests in
106+
`regression/cbmc/cpp` only. Another useful option is `-N`, which makes CTest
107+
list which tests it will run without actually running them.
108+
109+
110+
\subsubsection compilation-and-development-subsubsection-running-regression-tests-directly-with-test-pl Running regression tests directly with `test.pl`
111+
112+
In a directory corresponding to a tool or module, you can directly run a
113+
test directory as follows:
114+
115+
../test.pl -c PATH_TO_CBMC_FROM_DESC_FILE TEST_DIR
116+
117+
Note that `PATH_TO_CBMC_FROM_DESC_FILE` should either be absolute or be
118+
relative to the location of the test description files. If `TEST_DIR` is
119+
not provided then all test directories are run.
120+
87121

88122
\subsection compilation-and-development-subsection-unit-tests Unit tests
89123

@@ -98,6 +132,8 @@ To run the unit tests for JBMC, go to `jbmc/unit/` and run
98132

99133
../../<build-folder>/bin/java-unit
100134

135+
The unit tests are also included in CTest as `unit` and `java-unit`.
136+
101137
Note that some tests run which are expected to fail - see the summary at
102138
the end of the run to see how many tests passed, how many failed which were
103139
expected to and how many tests failed which were not expected to.

0 commit comments

Comments
 (0)