Re-exporting C symbols for cdylib

[qinsoon]

trafficstars

In an extern block, C symbols can be imported to Rust. When building Rust library into dylib, the symbols stay visible and can be used from C. However, when building into newly introduced cdylib, the imported C symbols are no longer visible. I am not aware of a mechanism to re-export the C symbols.

[petrochenkov]

Does making them pub help? (More precisely, making them nameable from outside of the crate.)

[qinsoon]

@petrochenkov Declaring pub doesn't help. I assume it is because pub does not mean they are accessible from native, thus their symbols can still be safely removed.

[retep998]

Can we please have some focus brought to this. For many applications which combine both C and Rust and are trying to create a cdylib, the lack of this feature is a blocker.

[michaelwoerister]

It seems clear that we need this functionality in one form or other. Ideally we'd have an RFC to determine who this should be implemented exactly. @aturon, how do we handle this, it being the impl period now? Or maybe we can decide on a straightforward way how to express this without an RFC.

[michaelwoerister]

I'll leave this nominated for now so it doesn't fall off the radar. Hopefully we'll have a clearer picture on how to proceed before the next dev-tools meeting.

[aturon]

I think we could approach the procedural issue here in a few ways:

1. We could land the functionality under a feature gate, and have an RFC prior to stabilization
2. We could have a public discussion on thread, involving lang team, with fcp signoff
3. We could just do an RFC :)

It's not like RFCs are prohibited per se, more that we're trying to focus the teams on executing already-planned work. This is small enough that it's probably not a big distraction.

I do think that in general this area of Rust could use more design focus, FWIW.

[retep998]

Currently the recommended way to export a function that you define yourself from a cdylib or staticlib is a #[no_mangle] pub extern fn foo(...) { ... } at the crate root. So I'd like to propose a solution that is as close to the existing forms as possible with minimal changes to the language:

• Declare the external functions directly at the crate root and make them public: extern { pub fn foo(...); }
• If the external function is declared elsewhere then publicly re-export into the crate root: pub use foo::bar;

The crate root being the root of the cdylib or staticlib crate itself.

[aturon]

cc @rust-lang/lang, adding this to our list of issues that need to be triaged.

[joshtriplett]

We discussed this one in the lang team meeting. This does need an RFC; how quickly that progresses will depend on how smoothly and non-bikesheddy the RFC goes. The approach of making symbols public if pub and #[no_mangle] seems reasonable.

Also, we should stop automatically exporting such symbols from a dylib, if possible. That may be difficult without regressions, but it's a serious library hygiene issue; it prevents hiding ABI changes for a dependency inside your own library, without breaking your own ABI.

[petrochenkov]

The approach of making symbols public if pub and #[no_mangle] seems reasonable.

Oh, this is great. What is done now (if I remember correctly), e.g. heuristic-based link-time visibility based on sum of factors like crate type, ABI (non-"Rust") and no_mangle even on private items is a mess and kinda breaks the spirit of Rust privacy (even if link-time visibility is a separate issue from compile-time one).

[joshtriplett]

I do think it'd be nice to have a broader RFC that explicitly defines when symbols should be exported, across all use-cases.

[dbrgn]

I just stumbled over this. Is there any workaround that we can use in the meantime?

In the worst case I could wrapp all 3rd party functions with my own:

extern crate saltyrtc_client_ffi;

pub type salty_event_loop_t = saltyrtc_client_ffi::salty_event_loop_t;

#[no_mangle]
pub extern "C" fn salty_event_loop_new() -> *mut salty_event_loop_t {
    saltyrtc_client_ffi::salty_event_loop_new()
}

...but I hope there's a better way than this.

If not, this is really a stumbling block when trying to reuse code from FFI compatible crates...

[Michael-F-Bryan]

I just encountered this issue at work. We've got a ffi_utils crate which contains general things like error handling, null pointer checks, and exception safety and I was hoping the original symbols would be exported from a cdylib which pulls in the ffi_utils crate.

@dbrgn for now I've found a really crummy workaround.... I'm creating a macro which will manually re-export the symbols. So you "only" need to call export_error_handling functions!() to make sure the symbols are included in the final DLL.

It feels horrible, but it works...

---

#[doc(hidden)]
#[macro_export]
macro_rules! export_c_symbol {
    (fn $name:ident($( $arg:ident : $type:ty ),*) -> $ret:ty) => {
        #[no_mangle]
        pub unsafe extern "C" fn $name($( $arg : $type),*) -> $ret {
            $crate::error_handling::$name($( $arg ),*)
        }
    };
    (fn $name:ident($( $arg:ident : $type:ty ),*)) => {
        export_c_symbol!(fn $name($( $arg : $type),*) -> ());
    }
}

/// As a workaround for rust-lang/rust#6342, you can use this macro to make sure
/// the symbols for `ffi_utils`'s error handling are correctly exported in your
/// `cdylib`.
#[macro_export]
macro_rules! export_error_handling_functions {
    () => {
    export_c_symbol!(fn clear_last_error());
    export_c_symbol!(fn last_error_length() -> ::libc::c_int);
    export_c_symbol!(fn last_error_length_utf16() -> ::libc::c_int);
    export_c_symbol!(fn error_message_utf8(buf: *mut ::libc::c_char, length: ::libc::c_int) -> ::libc::c_int);
    export_c_symbol!(fn error_message_utf16(buf: *mut u16, length: ::libc::c_int) -> ::libc::c_int);
    };
}

[dbrgn]

@Michael-F-Bryan thanks. yeah, there's probably currently no way around wrapping these functions.

by the way, you should try to avoid two #[no_mangle] functions with the same name, even across crate boundaries: https://users.rust-lang.org/t/including-third-party-no-mangle-functions-in-a-cdylib/15388/2

[Michael-F-Bryan]

by the way, you should try to avoid two #[no_mangle] functions with the same name, even across crate boundaries

Yep. The first time I ran the test suite with that macro I got lots of linker errors due to duplicate symbols. I "fixed" that by removing the #[no_mangle] from the original functions so the compiler would still mangle them.

Interestingly, I never got the "duplicate symbols" error when compiling the DLL in release mode. I'm assuming rustc/LLVM inlines the original functions and then strips out their now "unused" function symbols before they get to the linking stage.

[dbrgn]

It doesn't happen when linking together two libraries with rustc, but it happened to me when trying to link two independent shared libraries (compiled from Rust) into an iOS app. There might be a way to extract the Rust stdlib into a separate shared library (similar to the way it's done with libc, if I understand this correctly), but I haven't found a way to do this so far.

But that's probably offtopic here :)

[agalakhov]

Stepped into this issue as well. Unfortunately the solution with function wrapper does not work for me as I have to reexport arrays.

[tanriol]

The workaround does not work for me either as I need to (re)export variadic functions which cannot be defined in Rust.

[joshtriplett]

@tanriol While it doesn't address your immediate issue, in case you haven't already seen it, https://github.com/rust-lang/rfcs/pull/2137 and https://github.com/rust-lang/rust/issues/44930 may help you in the future.

[marmeladema]

I encountered this issue on Linux using rust 1.33.0. The trick to set lto to true and incremental to false works fine if the crate type is only cdylib, if you add rlib for example crate-type = ["cdylib", "rlib"] then no imported symbols are re-exported.

[DianaNites]

I just ran into, and spent hours, debugging this. :(

In my case I can trivially wrap it but it'd be nice if I could tell Rust not to throw away my symbol :(

[devsnek]

So right now it is not possible to re-export, for example, libc::malloc?

[pwnorbitals]

Bumped into it, spent more than a day troubleshooting this :( Do we have ideas on how to go forward on this issue ?

[samliddicott]

six years later.. ._@~·· rust has made me cry. I was promised a rose garden, but instead all my extern C were dropped.

(and that was after working around Cargo's beastly lack of feature forwarding that I had to hack around with RUSTFLAGS tunnelling --cfg feature=...)

[wxxedu]

Any updates on this issue?

[ellenhp]

I think (?) I may have found a workaround, if my understanding of this issue is correct. I have a project where I need to create a shared library that exports a bunch of no-mangle symbols that take C++ types as arguments (!). Not ideal for a lot of reasons, but the most salient one is that all of the rust tooling focused on exporting symbols from rust in a cdylib requires that the types be FFI-compatible.

What I ended up trying is including some C++ code that exports the symbols I needed, then calls into rust as-needed, but I immediately ran into this issue because the exported symbols in my C++ code were being hidden during the cdylib build at some point.

As it turns out, if you change your crate type to staticlib, the visibility of the C++ symbols does not change. I don't understand why, but I'm not about to question it. Once you have the static library you can turn it into a dynamic library. There may be a better way to do this but this is what I came up with.

Working example here: https://github.com/ellenhp/bambu-farm/tree/a87ad3988ef859355f86170f1371f1e94caa6615

Sorry if I'm misunderstanding the bug, but hopefully this helps someone.

[aidanhs]

For watchers of this issue, I've raised a relatively small RFC which covers a single narrow use-case: building a cdylib that wants to re-export a specific set symbols from a pre-built staticlib.

Note that this RFC doesn't actually propose adding any functionality on Linux (I don't know about the status on Windows) - you can already do this today by adding no_mangle to an item in an extern block, the RFC is to make this officially supported and remove the warning rustc emits.

There are a number of use-cases it doesn't cover, including:

1. re-exporting all symbols from a staticlib (workaround noted in RFC, same as https://github.com/rust-lang/rfcs/issues/2771#issuecomment-363695407 upthread)
2. types not representable by rust (workaround for most platforms noted in RFC, you can just ignore the types)
3. other binary/cdylib/staticlib linking combinations (workarounds for some combinations noted in RFC)

The above are valid scenarios, but require a little more work than this first step I'm trying to make. If you think that the raised RFC precludes any of these other use-cases, please comment over there.

@ellenhp that works fine, it just produces a cdylib larger than necessary - since a .a is just a collection of .o files, and .o files don't have symbol visibility, you've thrown away visibility information Rust provides to the linker to allow symbol elimination. You'll see that running nm -D on your cdylib lists a lot of Rust-related symbols. Aside from size, the main way this could cause an issue is if you have multiple versions of Rust linked into the same application (e.g. via different cdylibs) with visible symbols - there may be situations where the wrong functions are called, which is unlikely to end well.