Fuzz bindgen with C-Smith

[fitzgen]

C-Smith generates C programs for fuzzing C compilers. We should create infrastructure (scripts, etc) for running it against bindgen, that we check into tree. And we should run it against bindgen.

We should look into if C-Smith supports things like controlling the kinds of programs it generates. We don't care about functions' bodies generally, and we care a whole lot about different kinds of type definitions.

[fitzgen]

Note: #1033 added some initial infrastructure for fuzzing bindgen with C-Smith, but it appears not to be super useful yet.

Additional help tweaking knobs to get C-Smith to exercise bindgen a little better would be very welcome!

Also, perhaps seeing if running C-Smith for longer periods ends up uncovering anything.

Finally, it might make sense to file issues upstream with C-Smith if we can't tweak knobs to our satisfaction.

[pepyakin]

I'm wondering will it be useful to run rustc in check-only mode (as in cargo check) on bindings generated by the bindgen.

[fitzgen]

Yes, definitely. In fact, we should run the tests, so that we can assert size and alignment.

[e00E]

Are there any knobs we can tweak to make the types more complex?

In what kind of way would types get more complex anyway? In plain C they usually stay fairly simple.
The following command line switched of csmith seemed relevant:

--max-expr-complexity <num>: limit expression complexities to <num> (default 10).
--max-funcs <num>: limit the number of functions (besides main) to <num>  (default 10).
--max-struct-fields <num>: limit the number of struct fields to <num> (default 10).
--max-union-fields <num>: limit the number of union fields to <num> (default 5).

but I did not set them because I would not expect having more "things" is more complex from bindgen's perspective.

[e00E]

I just found a file that makes bindgen say
ERROR:bindgen::codegen::struct_layout: Calculated wrong layout for S5, too more 2 bytes
but it does create a full output rust file and does have exit code 0 (I tested other failures earlier which had exit code 1, so currently the script only looks for nonzero exit codes). If that means that bindgen will not always have a non zero exit code on failure then we will need to parse the output of stderr for lines start with ERROR , right?
input:

#pragma pack(push)
#pragma pack(1)
struct S5 {
   signed f0 : 11;
   unsigned f1 : 12;
   unsigned f2 : 23;
};
#pragma pack(pop)

output:

#[repr(C)]
#[derive(Debug, Copy)]
pub struct S5 {
    pub _bitfield_1: [u32; 2usize],
    pub __bindgen_align: [u8; 0usize],
}
... (tests and impls)

[e00E]

What is the best way to run the tests in the bindgen output? In a full project I would run cargo test but is there a better way?

[pepyakin]

I'm not sure whether this is best way, but you can try following command

bindgen [header.h] | rustc --test | ./rust_out

[e00E]

Looks like a test fails in the above example as well, so the new logic for checking if bindgen worked correctly would be:

• check for non zero exit code of bindgen
• check for any line of stderr starting with ERROR (or maybe even anything output to stderr, period)
• check for non zero exit code when executing the tests

[fitzgen]

Just found: #1034

Made a label for all issues we find with C-Smith: A-csmith.

Also cleaned up the driver.py, will send a PR in a second.

[e00E]

Ah, Im currenty changing it to follow the procedure outlined in my previous post, I will see what you changed.

[fitzgen]

Also, an idea: #1036

[pepyakin]

I'm wondering if there is a way to collect all known problems found by C-Smith and don't stop fuzzing if found any of them? 🤔

[fitzgen]

I'm wondering if there is a way to collect all known problems found by C-Smith and don't stop fuzzing if found any of them? 🤔

Not sure... maybe we just need to fix all the issues more quickly ;)