Quickchecking crate CLI #1177
Merged
Quickchecking crate CLI #1177
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
For sanity here are a couple fuzzed headers generated with the state of the current code. |
Prior to this commit the quickchecking crate used for generating proprty
tests
for bindgen was a [lib] target and had configurations that required
commenting/uncommenting code to enable/disable. This meant it was
inconvienent/prohibitive to configure the property tests on a per-run
basis.
This commit reorganizes the `quickchecking` crate to provide both [lib]
and
[[bin]] targets in order to expose those configurations through a CLI.
The configurations that are exposed through the [[bin]] target's CLI
are:
* Count/number of tests to run.
* Directory to provide fuzzed headers
* Generation range corresponding to the range quickcheck uses to
* generate arbitrary.
__Usage from the__ `tests/quickchecking` __directory__
```bash
quickchecking 0.2.0
Bindgen property tests with quickcheck. Generate random valid C code and
pass it to the csmith/predicate.py script
USAGE:
quickchecking [OPTIONS]
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
OPTIONS:
-c, --count <COUNT> Count / number of tests to run. Running a
fuzzed header through the predicate.py script can
take a long time, especially if the
generation range is large. Increase this number if you're
willing to wait a while. [default: 2]
-p, --path <PATH> Optional. Preserve generated headers for
inspection, provide directory path for header
output. [default: None]
-r, --range <RANGE> Sets the range quickcheck uses during
generation. Corresponds to things like arbitrary usize
and arbitrary vector length. This number
doesn't have to grow much for that execution time to
increase significantly. [default: 32]
```
Because the actual work of running the property tests moved to the
[[bin]]
target, rather than duplicate that code in the `quickchecking` crate's
tests
directory, some actual (very basic) tests for the `quickchecking` crate
were
added.
*Note: I'm not attached to any of the option flags, if there are better
characters/words for any of the options I've exposed I'll be happy to
revise!
Thanks for taking a look, looking forward to feedback!
Closes #1168
r? @fitzgen
fitzgen
approved these changes
Dec 5, 2017
There was a problem hiding this comment.
Looks great!
Now that the tests are about the quickchecking crate itself, rather than actually quickchecking bindgen, we should run cargo test in CI instead of cargo check.
Once you add that, you can comment @bors-servo r=fitzgen and the PR will get tested and merged.
Thanks!
|
@bors-servo delegate+ |
|
✌️ @Snewt can now approve this pull request |
|
@fitzgen ah awesome, can do (this evening). |
The changes reflected in this PR include the logic to test the `quickcheking` crate itself. Rather that just validate that the `quickchecking` crate builds in CI with `cargo check`, we can run now `cargo test`.
|
@bors-servo r=fitzgen |
|
📌 Commit 2c77652 has been approved by |
bors-servo
pushed a commit
that referenced
this pull request
Dec 6, 2017
Quickchecking crate CLI
Prior to this commit the quickchecking crate used for generating proprty tests for bindgen was a [lib] target and had configurations that required commenting/uncommenting code to enable/disable. This meant it was inconvienent/prohibitive to configure the property tests on a per-run basis. This commit reorganizes the `quickchecking` crate to provide both [lib] and [[bin]] targets in order to expose those configurations through a CLI.
The configurations that are exposed through the [[bin]] target's CLI are:
* Count/number of tests to run.
* Directory to provide fuzzed headers
* Generation range corresponding to the range quickcheck uses to
* generate arbitrary.
__Usage from the__ `tests/quickchecking` __directory__
```bash
quickchecking 0.2.0
Bindgen property tests with quickcheck. Generate random valid C code and pass it to the csmith/predicate.py script
USAGE:
quickchecking [OPTIONS]
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
OPTIONS:
-c, --count <COUNT> Count / number of tests to run. Running a fuzzed header through the predicate.py script can
take a long time, especially if the generation range is large. Increase this number if you're
willing to wait a while. [default: 2]
-p, --path <PATH> Optional. Preserve generated headers for inspection, provide directory path for header
output. [default: None]
-r, --range <RANGE> Sets the range quickcheck uses during generation. Corresponds to things like arbitrary usize
and arbitrary vector length. This number doesn't have to grow much for that execution time to
increase significantly. [default: 32]
```
Because the actual work of running the property tests moved to the [[bin]]
target, rather than duplicate that code in the `quickchecking` crate's tests
directory, some actual (very basic) tests for the `quickchecking` crate were
added.
*Note: I'm not attached to any of the option flags, if there are better characters/words for any of the options I've exposed I'll be happy to revise!
Also, I'm not sure how palatable the "global singleton" is for managing context (output path) across tests in the `lib.rs` file. Very open to suggestions on how to manage that if it's not an acceptable approach.
Thanks for taking a look, looking forward to feedback!
Closes #1168
r? @fitzgen
|
☀️ Test successful - status-travis |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to this commit the quickchecking crate used for generating proprty tests for bindgen was a [lib] target and had configurations that required commenting/uncommenting code to enable/disable. This meant it was inconvienent/prohibitive to configure the property tests on a per-run basis. This commit reorganizes the
quickcheckingcrate to provide both [lib] and [[bin]] targets in order to expose those configurations through a CLI.The configurations that are exposed through the [[bin]] target's CLI are:
Usage from the
tests/quickcheckingdirectoryquickchecking 0.2.0 Bindgen property tests with quickcheck. Generate random valid C code and pass it to the csmith/predicate.py script USAGE: quickchecking [OPTIONS] FLAGS: -h, --help Prints help information -V, --version Prints version information OPTIONS: -c, --count <COUNT> Count / number of tests to run. Running a fuzzed header through the predicate.py script can take a long time, especially if the generation range is large. Increase this number if you're willing to wait a while. [default: 2] -p, --path <PATH> Optional. Preserve generated headers for inspection, provide directory path for header output. [default: None] -r, --range <RANGE> Sets the range quickcheck uses during generation. Corresponds to things like arbitrary usize and arbitrary vector length. This number doesn't have to grow much for that execution time to increase significantly. [default: 32]Because the actual work of running the property tests moved to the [[bin]]
target, rather than duplicate that code in the
quickcheckingcrate's testsdirectory, some actual (very basic) tests for the
quickcheckingcrate wereadded.
*Note: I'm not attached to any of the option flags, if there are better characters/words for any of the options I've exposed I'll be happy to revise!
Also, I'm not sure how palatable the "global singleton" is for managing context (output path) across tests in the
lib.rsfile. Very open to suggestions on how to manage that if it's not an acceptable approach.Thanks for taking a look, looking forward to feedback!
Closes #1168
r? @fitzgen