Add property based testing with quickcheck #970
Labels
Comments
This was referenced Nov 14, 2017
bors-servo
pushed a commit
that referenced
this issue
Nov 30, 2017
Property testing with quickcheck This PR represents an attempt to address issue #970. It also represents a portion of the meta issue for fuzzing #972. The code base reflected here uses quickcheck to generate C headers that include a variety of types including basic types, structs, unions, function prototypes and function pointers. The headers generated by quickcheck are passed to the `csmith-fuzzing/predicate.py` script. Examples of headers generated by this iteration of the tooling can be viewed [here](https://gist.github.com/snewt/03ce934f35c5b085807d2d5cf11d1d5c). At the top of each header are two simple struct definitions, `whitelistable` and `blacklistable`. Those types are present in the vector that represents otherwise primitive types used to generate. They represent a naive approach to exposing custom types without having to intuit generated type names like `struct_21_8` though _any actual whitelisting logic isn't implemented here_. Test success is measured by the success of the `csmith-fuzzing/predicate.py` script. This means that for a test to pass the following must be true: - bindgen doesn't panic - the resulting bindings compile - the resulting bindings layout tests pass #### Usage ```bash cd tests/property_test cargo test ``` Some things I'm unsure of: #### Where should this feature live? At the moment it lives in `tests/property_test` but isn't run when `cargo test` is invoked from bindgen's cargo manifest directory. #### What's an acceptable ammount of time for these tests to take? At this point, the source is genereated in ~1 second but the files are large enough that it takes the `predicate.py` script ~30 seconds to run through each one. In order for the tests to run in under a minute only 2 are generated by quickcheck by default. This can be changed in the `test_bindgen` function of the `tests/property_test/tests/fuzzed-c-headers.rs` file. #### How do we expose the generated code for easy inspection? For now the `run_predicate_script` function in the `tests/property_test/tests/fuzzed-c-headers.rs` file contains a commented block that will copy generated source in the `tests/property_test/tests` directory. Should it be easier? #### Special casing There is some logic in the fuzzer that disallows 0 sized arrays because tests will regulary fail due to issues documented in #684 and #1153. Should this be special casing? #### Does the fuzzer warrant its own crate? After any iterations the reviewers are interested in required to make this a functional testing tool, should/could the fuzzing library be made into its own crate? I didn't move in that direction yet because having it all in one place seemed like the best way to figure out what works an doesn't but I'm interested in whether it might be useful as a standalone library. #### What does it look like to expose more useful functionality? I'm looking forward to feedback on how to make this a more useful tool and one that provides the right configurability. Thanks! r? @fitzgen
This was referenced Nov 30, 2017
Closed
|
@fitzgen Any opinion on the status of this issue? "property based testing with quickcheck" exists now but there's still room for expanding the scope. Maybe tests for whitelisting and opaque types should be added before closing? Maybe some generated C++? |
|
I think we can close this issue now -- thanks for all your work thus far! |
craftloops
referenced
this issue
in shnewto/sheas.blog
Jan 5, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://github.com/BurntSushi/quickcheck
Another kind of fuzzing of sorts. We should use
quickcheckto generate struct definitions and whitelisting/opaque/etc rules, generate a header containing those definitions in C source form, runbindgenon the header, and then assert various properties about the resulting bindings.Ideas of properties to assert:
bindgendoesn't panicAll of these rely on generating valid C headers, not just garbage, which is a little bit of work, but seems do-able.
The text was updated successfully, but these errors were encountered: