quickhecking crate CLI #1168
Comments
bors-servo
pushed a commit
that referenced
this issue
Dec 6, 2017
Quickchecking crate CLI
Prior to this commit the quickchecking crate used for generating proprty tests for bindgen was a [lib] target and had configurations that required commenting/uncommenting code to enable/disable. This meant it was inconvienent/prohibitive to configure the property tests on a per-run basis. This commit reorganizes the `quickchecking` crate to provide both [lib] and [[bin]] targets in order to expose those configurations through a CLI.
The configurations that are exposed through the [[bin]] target's CLI are:
* Count/number of tests to run.
* Directory to provide fuzzed headers
* Generation range corresponding to the range quickcheck uses to
* generate arbitrary.
__Usage from the__ `tests/quickchecking` __directory__
```bash
quickchecking 0.2.0
Bindgen property tests with quickcheck. Generate random valid C code and pass it to the csmith/predicate.py script
USAGE:
quickchecking [OPTIONS]
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
OPTIONS:
-c, --count <COUNT> Count / number of tests to run. Running a fuzzed header through the predicate.py script can
take a long time, especially if the generation range is large. Increase this number if you're
willing to wait a while. [default: 2]
-p, --path <PATH> Optional. Preserve generated headers for inspection, provide directory path for header
output. [default: None]
-r, --range <RANGE> Sets the range quickcheck uses during generation. Corresponds to things like arbitrary usize
and arbitrary vector length. This number doesn't have to grow much for that execution time to
increase significantly. [default: 32]
```
Because the actual work of running the property tests moved to the [[bin]]
target, rather than duplicate that code in the `quickchecking` crate's tests
directory, some actual (very basic) tests for the `quickchecking` crate were
added.
*Note: I'm not attached to any of the option flags, if there are better characters/words for any of the options I've exposed I'll be happy to revise!
Also, I'm not sure how palatable the "global singleton" is for managing context (output path) across tests in the `lib.rs` file. Very open to suggestions on how to manage that if it's not an acceptable approach.
Thanks for taking a look, looking forward to feedback!
Closes #1168
r? @fitzgen
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
quickcheckingcrate, for property testing with fuzzed C headers, is a library at the moment. If instead it was a binary application some configuration could be exposed through a CLI.(crate lives in the tests directory here: https://github.com/rust-lang-nursery/rust-bindgen/tree/master/tests/quickchecking)
Some configuration that may be worth exposing:
Number of tests to run
Each test on a fuzzed header takes about ~30 to run through the
csmith-fuzzing/predicate.pyscript. In order to run in under ~1 minute, the default is to only generate and run 2 tests. It would be nice if that was easier to toggle.Generate range
The range that quickcheck uses during generation is configurable. This range corresponds to things like arbitrary usize and arbitrary vector length. Currently this value is also defaulting to cater to a reasonable execution time. This number doesn't have to grow much for that execution time to increase significantly.
Provide generated headers for inspection
Default behavior is to generate temporary fuzzed headers in a directory decided by the
TempDircrate as well as the quickchecking crate's tests directory. Providing/disabling the visibility of that code requires commenting/uncommenting some code in the fuzzed-c-headers.rs test file. It'd probably be nice to make that easier.Corresponds to extending #970
The text was updated successfully, but these errors were encountered: