Problems binding to Resonance Audio - failing cargo layout tests, methods generated as free functions, probable struct return problem

[Neurrone]

Hi,

I'm attempting to create bindings for Resonance audio. Here are some things I've noticed, which I'm not sure might be part of the cause:

1. When running cargo test, there are numerous failures, and I noticed the failing test cases are either:

• instantiations of template functions
• Classes inheriting from parents that in turn, inherit from someone else. The parent class has implementation of some of the grandparent's methods, with some virtual.

2. Most methods in c++ are generated as free-standing functions that take a this pointer in Rust. For example,

int vraudio::ResonanceAudioApiImpl::CreateStereoSource(size_t num_channels)

comes out as

pub unsafe extern "C" fn ResonanceAudioApiImpl_CreateStereoSource(
    this: *mut c_void, 
    num_channels: usize
) -> ResonanceAudioApi_SourceId

type ResonanceAudioApi_SourceId = c_int;

3. It crashes when after I create a class, I attempt to call a method on it (src/main.rs). As discussed in the previous issue, the most likely cause is issues with ABIs and struct return types rust#38258

[emilio]

Classes inheriting from parents that in turn, inherit from someone else. The parent class has implementation of some of the grandparent's methods, with some virtual.

That's #380 I think.

A standalone test-case that reproduces the issues would be much appreciated.

Most methods in c++ are generated as free-standing functions that take a this pointer in Rust. For example.
[...]
It crashes when after I create a class, I attempt to call a method on it (src/main.rs). As discussed in the previous issue, the most likely cause is rust-lang/rust#38258

Which OS is this? How do you create a class? I don't think such simple method should crash.

[Neurrone]

MSVC x64, windows 10 1803. The arguments to both the constructor and source creation function are also very simple - just ints. Maybe the other possibility is a problem with how the constructor from Rust is defined.

// from src/main.rs
use resonance_audio_sys::ResonanceAudioApiImpl as api;
use std::ffi::c_void;

fn main() {
    let mut resonance_api = unsafe {
        api::new(2, 100, 44100);
    };
    println!("Resonance audio api created.");
    // crashes here
    let source = unsafe { resonance_audio_sys::ResonanceAudioApiImpl_CreateStereoSource(&mut resonance_api as *mut _ as *mut c_void, 2) };
    println!("Source created.");
}

[Neurrone]

I'll also test the equivalent code in C++ just to be 100% sure.

[emilio]

Yeah, IIRC the reason we don't generate methods on MSVC is because they use the thiscall ABI, which rustc doesn't support anywhere except nightly. Can you check with rust_target(Nightly)? Does that fix the issue?

[emilio]

That should allow you to call resonance_api.CreateStereoSource(3) too.

[Neurrone]

Hm, I think I'm already using nightly. In build.rs, I have

.rust_target(bindgen::RustTarget::Nightly)

Is there something else I need to do to enable it? I'm on the latest nightly currently, MSVC 2017.8.

[emilio]

Hmm, that's weird, we allow generating thiscall methods if supported. This is tested in win32-thiscall_nightly.{h,rs}.

Is that method virtual by any chance, or something like that? May that explain that / the crash?

Again, a test-case that repros this in a standalone manner is the best way to investigate.

[Neurrone]

@emilio I'll try installing creduce using the windows subsystem for linux, once the apt-get update completes. It seems to be downloading at 56kbps speeds for some reason.

[Neurrone]

I have the following in the .sh file:

python3 ./csmith-fuzzing/predicate.py \
    --bindings-grep BufferedSourceNode \
    --expect-layout-tests-fail \
    --layout-tests-grep "thread 'bindgen_test_BufferedSourceNode' panicked" \
    ./isolated-test-case.h

and am invoking it in the same directory where I cloned bindgen. I've also placed __bindgen.cpp and __bindgen.ii but it still doesn't work. It says it can't find the .h file.

Then I tried pointing it to __bindgen.ii instead, and there were dozens of compiler errors.

[Neurrone]

While trying to run cargo test in the tests/expectation folder to test thiscall:

   Compiling tests_expectations v0.1.0 (D:\source\rust-bindgen\tests\expectations)
error: unknown character escape: D
 --> D:\source\rust-bindgen\tests\expectations\target\debug\build\tests_expectations-18b98ff00f0177ae\out/libclang_version_specific_generated_tests.rs:2:15
  |
2 | #[path = "\\?\D:\source\rust-bindgen\tests\expectations\tests\libclang-3.8\abi_variadic_function.rs"]
  |               ^

error: unknown character escape: s
 --> D:\source\rust-bindgen\tests\expectations\target\debug\build\tests_expectations-18b98ff00f0177ae\out/libclang_version_specific_generated_tests.rs:2:18
  |
2 | #[path = "\\?\D:\source\rust-bindgen\tests\expectations\tests\libclang-3.8\abi_variadic_function.rs"]
  |                  ^

error: unknown character escape: e
 --> D:\source\rust-bindgen\tests\expectations\target\debug\build\tests_expectations-18b98ff00f0177ae\out/libclang_version_specific_generated_tests.rs:2:44
  |
2 | #[path = "\\?\D:\source\rust-bindgen\tests\expectations\tests\libclang-3.8\abi_variadic_function.rs"]

...hundreds more like this

[emilio]

Looks like we need to normalize the path here:

https://github.com/rust-lang-nursery/rust-bindgen/blob/6fc0a31febb63d77da1a38aa2eea9d10fbea0d0d/tests/expectations/build.rs#L50

But not sure what's the format rust expects or what not.

[Neurrone]

Do those tests compile fine in Linux?

[emilio]

Yes

[emilio]

(We run them on automation)

[emilio]

But on linux the path separator is of course different. It may be a matter of changing that path.display() line by: format!("{}", path.display()).replace("\\", "\\\\").

[Neurrone]

That appears to do the trick, but now I get the following errors. Pasting all 12 errors which are the same, just different occurances in the same file, in case source code changes are needed to fix.

I'm on a 64 bit machine, I wonder if this is caused by a nightly bug.

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:125:18
    |
125 |         unsafe { ::std::mem::transmute(self._bitfield_1.get(0usize, 64u8) as u64) }
    |                  ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u64 (64 bits)
    = note: target type: u32 (32 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:130:28
    |
130 |             let val: u64 = ::std::mem::transmute(val);
    |                            ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:136:18
    |
136 |         unsafe { ::std::mem::transmute(self._bitfield_1.get(64usize, 64u8) as u64) }
    |                  ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u64 (64 bits)
    = note: target type: u32 (32 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:141:28
    |
141 |             let val: u64 = ::std::mem::transmute(val);
    |                            ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:147:18
    |
147 |         unsafe { ::std::mem::transmute(self._bitfield_1.get(128usize, 1u8) as u64) }
    |                  ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u64 (64 bits)
    = note: target type: u32 (32 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:152:28
    |
152 |             let val: u64 = ::std::mem::transmute(val);
    |                            ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:158:18
    |
158 |         unsafe { ::std::mem::transmute(self._bitfield_1.get(192usize, 64u8) as u64) }
    |                  ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u64 (64 bits)
    = note: target type: u32 (32 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:163:28
    |
163 |             let val: u64 = ::std::mem::transmute(val);
    |                            ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:177:44
    |
177 |             let m_bitfield: u64 = unsafe { ::std::mem::transmute(m_bitfield) };
    |                                            ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:181:39
    |
181 |             let m_bar: u64 = unsafe { ::std::mem::transmute(m_bar) };
    |                                       ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:185:37
    |
185 |             let foo: u64 = unsafe { ::std::mem::transmute(foo) };
    |                                     ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error[E0512]: transmute called with types of different sizes
   --> tests\issue-739-pointer-wide-bitfield.rs:189:37
    |
189 |             let bar: u64 = unsafe { ::std::mem::transmute(bar) };
    |                                     ^^^^^^^^^^^^^^^^^^^^^
    |
    = note: source type: u32 (32 bits)
    = note: target type: u64 (64 bits)

error: aborting due to 12 previous errors

[emilio]

Are you compiling with a 32-bit target by any chance?

[emilio]

cargo test should probably generate working code and some differences in your machine if so, I'd expect.

[emilio]

(on the root bindgen directory).

[Neurrone]

Nope, unless rust-bindgen itself is configured to. I have only the x86-64 (64-bit) toolchain installed.

[Neurrone]

Running cargo test from the bindgen root directory works fine - everything passes.

[Neurrone]

I've just tried defining a struct, and attempting to transmute its address into a u32, and it fails, as expected.

So I think its something to do with those specific files Bindgen has specifically.

[emilio]

Hmm, can it be that the rust c_ulong and the C c_ulong disagree?

What does sizeof(unsigned long) return in your machine? What about _Alignof(unsigned long)?

[Neurrone]

I'm not at my computer currently, but based on this page after some googling and my own memory, I'm almost certain that longs are 32-bit, even on 64 bit windows, and that you need long long for 64-bit integers. I'll test once I'm back.

[Neurrone]

On MSVC 2017.8.2, x86 and x64.

source

#include <stdio.h>

int main(int argc, char* argv) {
    long i = 5;
    unsigned long j = 6;
    printf("Size of long: %d, alignment: %d", sizeof(i), alignof(long));
    printf("Size of unsigned long: %d, alignment: %d", sizeof(j), alignof(unsigned long));
    return 0;
}

output

Size of long: 4, alignment: 4Size of unsigned long: 4, alignment: 4

[Neurrone]

Is there a way for me to only run the win32 nightly.rs test file to circumvent these errors?

Also, how do I use the __bindgen.ii file with predicate.sh to get a reduced test case?

[Neurrone]

Oh god, I feel so stupid.

    let mut resonance_api = unsafe {
        api::new(2, 100, 44100);
    };

I only noticed the extra ; after the call to new completely by chance. Hence, api has the type (). Removing the ; fixed it.

I'm not sure how conversion is defined to (), but I suspect that manipulating its address is undefined behaviour or something.

[Neurrone]

I ran into another access violation when trying to call some other methods on this class, even though the layout test for this class passes. The method does create instances of objects which have failing layout tests, so I'm not sure if that might be the cause.

In this case, I was able to work around it by having some glue code in C++ which returns a pointer to an instance of this object on the heap, and using that as the this pointer.

[emilio]

If you could reduce the test-case for the layout failures I'd be interested in taking a look when I have the time.

So sorry for the lag in replying here, too :(

[Neurrone]

How do I use the __bindgen.ii file with predicate.sh to get a reduced test case?