C++ ABI in MSVC and function returning non-POD type

[rodrigorc]

I am happily using bindgen to link Rust to a C++ library, and it works great... except when compiling for the MSVC target (that is i686-pc-windows-msvc or x86_64-pc-windows-msvc), that my program crashes badly with random exceptions.

I have reduced the faulty code to the code in this sample repo, ready to run:

https://github.com/rodrigorc/testabi

Input C/C++ Header

There are two types, for reference, Ok works fine, Err fails. The only difference is the constructor, that IIUIC makes the type non-POD and changes the ABI.

struct Ok
{
    int x;
};

Ok TestOk()
{
    Ok r;
    r.x = 1;
    return r;
}

////////////////////////////

struct Err
{
    Err() {} // <--- Difference here!!!
    int x;
};

Err TestErr()
{
    Err r;
    r.x = 1;
    return r;
}

Bindgen Invocation

It fails both with clang and msvc compilers.

    let bindings = bindgen::Builder::default()
        .header("test.cpp")
        .generate()
        .expect("bindings gen failed");
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("Couldn't write bindings!");

    let mut build = cc::Build::new();
    //build.compiler("clang");
    build.file("test.cpp");
    build.compile("test");

Actual Results

When running the program in windows, natively compiled, it prints garbage:

[src/main.rs:5:5] s = Ok {
    x: 1,
}
[src/main.rs:8:5] t = Err {
    x: 155599256,
}

Or crashes with something like:

error: process didn't exit successfully: `target\debug\testabi.exe` (exit code 0xc0000005, STATUS_ACCESS_VIOLATION)

Or sometimes other less known exceptions. I reckon the stack is getting corrupted.

Expected Results

When cross compiling from Linux, or running natively on Linux, it prints the expected:

[src/main.rs:5:5] s = Ok {
    x: 1,
}
[src/main.rs:8:5] t = Err {
    x: 1,
}

[rodrigorc]

After some extra investigation, it looks like MSVC has two kinds of return structs: small and big.

The "small" return ABI is used for C structs that have 8 bytes or fewer, and the value is returned in registries.

The "big" return ABI is used for structs greater than 8 bytes or those with C++ parts (constructors, destructors, vtables, base members...).

I think that the relevant code in CLang would be:

https://github.com/llvm/llvm-project/blob/45fc655b9eb038a8daf739b9dafb46fe0aac2d60/clang/lib/CodeGen/MicrosoftCXXABI.cpp#L1168

The issue here is that in Rust there is no way to choose which one you want. When you write #[repr(C)] into a struct it will use the small or big ABI depending on the size of the type, not caring about the C++ pieces.

So a C++ struct with sizeof <= 8 and a constructor cannot be expressed as a FFI-to-MSVC type in current Rust.

I'm not sure what Bindgen should do here. The easiest solution would be just not to emit bindings for such a function. I've tried looking at the code, but came back empty handed...

[sagudev]

We have some wrappers for this in servo/mozjs: https://github.com/servo/mozjs/blob/bf41ed080ec3c2fb84a53928c7db7691b0a0b161/mozjs-sys/src/jsglue.cpp#L997