rust-lang
RFC: `#[export]` (dynamically linked crates)
This generally looks great and useful, but there's one aspect that seems a bit strange to me. From my understanding, to link to such a shared crate you would need to have either (a version of) its source code available or some kind of types-and-functions-without-function-bodies version of it (i.e. something like the Rust version of a C header file).
Did you consider including the API of the crate in a way that rustc can understand inside the shared library, e.g. in a separate section, or as a separate interface definition file? That way the shared library build results would be all that is necessary to link to the library, and for example cargo (and other build systems) could later get a mechanism to look for system installed shared libraries in that format and directly make use of them.
@sdroege That's certainly a possibility and not incompatible with this RFC. The "future possibilities" section already mentions the possibility of adding more information in a separate (optional / debug) section, and what you're proposing seems to be a form of that.
something like the Rust version of a C header file
Yes, I'm basically proposing that we can have #[export] attributes rather than a separate header file (like C) or a bindings crate (Rust today).
Sounds good to me, thanks for the fast response :) Maybe you could extend the "A tool to create a stripped, 'dynamic import only' version [...]" item in the "future possibilities" section with a sentence that this could also be as a binary format as part of a shared library section or so.
Such a mechanism would also be useful to allow non-Rust languages to link to such shared libraries without having to parse Rust code or requiring tools for converting the API definitions between any possible source and target language.
How about flipping the default and exporting all the same symbols as static linking does, and provide an opposite attribute to opt out? (#[unstable_abi] or such)
I imagine once Rust ABI situation matures, users will want to support both kinds of linkage without having to manage the public API twice, so the explicit opt in useful initially may be a liability long term. Similarly Rust was conservative with #[must_use] which causes clutter and is inconsistently applied, unlike Swift's discatdableResult.
@kornelski you can do that by adding a #![export] once globally to your crate. It works recursively over pub items, as explained in the RFC.
For the forseeable future, we're not going to be able to export fully generic functions (e.g. like std::option::Option::map), so I doubt most crates would be using this for their entire interface.
Also, for any type with invariants (types with private fields), we can't safely mark those as exported, without the author unsafely committing to keeping the invariants stable. (See the section on private types in the RFC.)
I imagine once Rust ABI situation matures, users will want to support both kinds of linkage without having to manage the public API twice
I think that's inevitable. Basically, pub is about an API commitment, e.g. File::open(path) always existing, while #[extern] is about an ABI commitment, e.g. File always being a wrapper around a file descriptor. We can't really change pub to suddenly also include the ABI commitment, which includes committing to not changing any invariants about the private fields.
As Prior Art, this could reference Swift's ABI attributes. In particular,
@frozen seems equivalent to #[export] #[repr(Swift)] on structs (for the appropriate definition of #[repr(Swift)].
@usableFromInline seems equivalent to #[export] on functions, except that it is allowed on internal/pub(crate) functions, to allow use from @inlinable functions.
@inlinable seems equivalent to the #[export_inline] future direction on functions, except that it is allowed on internal/pub(crate) functions, to allow use from other (public) @inlinable functions.
In addition, IIRC, swift defaults public functions in ABI stable libraries to ABI stable, but the Swift stdlib uses the unstable @_alwaysEmitIntoClient to unstably inline functions to prevent ABI presence.
While I'm really happy to see some progress on the ABI front, I feel like this still doesn't address the most painful point of building shared libraries in Rust: "What if some of my dependencies didn't brand their types with #[export]?". We already have ways to get a stable ABI for types we fully own (C-repr, abi_stable, stabby...), but as soon as dependencies are introduced, things get real ugly real fast.
I really wish we'd put the option on the table of having the compiler "contaminate" types that are used by #[export]ed symbols. While this doesn't solve the "mismatching library version" part of the problem, there are existing ways to check for this problem. There are no ways to force a dependency to use a stable layout for some of its types.
One of the main use-cases for dynamic loading of libraries is for plugin-like systems, where you might have multiple libraries exposing the same API.
This RFC does not appear to be designed to solve this use-case, but it solves several closely related problems, and it would be a shame if the plugin use-case required a totally orthogonal approach.
Is there a way this RFC could be naturally built upon to support such a feature?
Maybe I misunderstood something. Why do you limit the RFC to shared libraries? Static libraries are sometimes also useful. Ok, it is perhaps only a special case. But we develop a library operating systems and link a static library as kernel to the application. Currently, the interface between application and libos is a C interface....
So, I have a feeling this was explicitly left out of the RFC, but it feels worth mentioning something about the intended loading mechanism for these dynamic crates.
I think it is (mostly implied) that cargo run will work out-of-the-box even with these settings, although it would be good to mention in the RFC explicitly as well. It would also be useful to clarify if this enables shared storage of dynamically-compiled crates too, although I would understand initially tightening this down and deferring that as well.
Obviously, loading dynamic code is complicated and each operating system has its own way of dealing with things. There are "system-wide" locations as well as "app-specific" locations that will be most suited further down the road. However, I'm not 100% sure how exactly those would fit into this current proposal. The most obvious mechanism currently is that used by cargo install, but the RFC also doesn't say anything about whether that would work with this mechanism. I don't think this should be required for this proposal, but I think it should at least have some sort of opinion on the path there, and what kinds of steps would need to be taken to design that. I don't think end users should have to operate differently to cargo install a crate that's dynamically linked, but perhaps this would also lead to some notion of cargo installing library crates too? It all seems a bit fuzzy.
Why would cargo run do anything like an install / put things in a global location? I would assume that cargo run on an executable with a dynamic dependency would just put the executable in target/debug/myprog and the .dll or .so for the library in target/debug/deps/mylib.so.
Why would
cargo rundo anything like an install / put things in a global location? I would assume thatcargo runon an executable with a dynamic dependency would just put the executable intarget/debug/myprogand the.dllor.sofor the library intarget/debug/deps/mylib.so.
This isn't very out-there to consider when you remember that cargo will globally cache crate metadata and information from registries. What's to assume that dynamically built crates wouldn't be put there as well, since they by design could be used across different builds?
For example, I have eight different versions of the serde crate source in my .cargo/registry folder when they could have been copied into individual target folders, but aren't because they wouldn't change across workspaces that use them. The same could happen for dynamic library builds too.
Maybe I misunderstood something. Why do you limit the RFC to shared libraries? Static libraries are sometimes also useful. Ok, it is perhaps only a special case. But we develop a library operating systems and link a static library as kernel to the application. Currently, the interface between application and libos is a C interface....
That was also brought up here: https://github.com/rust-lang/rfcs/pull/3435#discussion_r1202505606:
That's a good point. Perhaps that means we should use slightly different names, but I don't think that changes anything else substantially about the RFC.
Is it possible to export an unsafe fn?
I think that has some similar issues to structs with invariants: even if the signature of the function stays the same, its safety requirements might change, therefore somehow those requirements have to be encoded into the symbol name.
A few questions:
- Will it be possible to do
#[export, no_mangle](or separate attributes)? To create one interface and link it from both C and Rust, this would only do the "treat dependencies as header files" part (i.e. layout & symbols only, no codegen). Maybe this could be a good minimum viable product since it punts the mangling question that is still under heavy discussion?- I would be happy as an even smaller step if
rustcwith--extern=dynamic=...or equivalent would be able to enable this header-like behavior for all publicextern "C"functions even without the#[export]attribute to make all existing Rustcdylibs usable as.sowithout duplicating interfaces. - Based on this, there seems to be two separate concerns blended in this RFC. Concern 1 being, we need this "header file" behavior to have rust-links-rust work ergonomically, even if it goes through the
no_mangle+extern "C"that is available today. I don't think#[export]needs to have anything to do with this part. Concern 2 is that we want a way to ensure synchronization of Rust type ABIs via mangling, which I believe is what#[export]will do
- I would be happy as an even smaller step if
- How heavily will real-world usage depend on Cargo supporting binary dependencies https://github.com/rust-lang/cargo/issues/9096? Right now that is kind of a blocker to ergonomic rust shared libraries (at least for my use case)
- How does this interact with
dylibvs.cdylib? I assume that any prepackaged artifacts generally need to becdylibin order to ensure stability across different compiler versions, but the interaction withlibstdis a bit weird here (the differences between these two interfaces isn't very well documented) - Is this RFC more or less looking for team review at this point? Or are there large changes / blockers expected
There is also some overlap with how we link in dependencies when we create staticlibs (the duplication problem) but this is being worked on separately
Update: We're working on launching a research project around the problem this RFC is trying to solve. See https://github.com/rust-lang/rust-project-goals/pull/155
Will it be possible to do
#[export, no_mangle](or separate attributes)? To create one interface and link it from both C and Rust, this would only do the "treat dependencies as header files" part (i.e. layout & symbols only, no codegen). Maybe this could be a good minimum viable product since it punts the mangling question that is still under heavy discussion?
If we allow both attributes, I'd expect the result would be that the item will be available under two symbol names. (The export mangled one, and the no_mangle unmangled one.)
It sounds like you're asking if we could do #[extern] without mangling. I think the answer is a no, as the main point of this RFC is to achieve safety through symbol mangling.
Nonetheless it's an interesting question what would remain of this RFC if we postponed that part for an initial implementation.
How heavily will real-world usage depend on Cargo supporting binary dependencies Tracking Issue for RFC 3028: Allow "artifact dependencies" on bin, cdylib, and staticlib crates cargo#9096? Right now that is kind of a blocker to ergonomic rust shared libraries (at least for my use case)
We'd ideally need cargo to understand the concept of a dynamically linked dependency to make this an ergonomic feature. However, using (generated?) 'header-only' crates, we can make this ergonomic enough for a first usable version without any special cargo support.
How does this interact with
dylibvs.cdylib? I assume that any prepackaged artifacts generally need to becdylibin order to ensure stability across different compiler versions, but the interaction withlibstdis a bit weird here (the differences between these two interfaces isn't very well documented)
Yup, the interaction of different libstds is an unsolved problem. The easiest solution is to use 'cdylib', but we will need to address this problem if we ever expect to seemlessly transfer types like String through extern boundaries, which could be allocated on one side and dropped on another.
Is this RFC more or less looking for team review at this point? Or are there large changes / blockers expected
I still need to process all the feedback and update the RFC. But those are mostly relatively minor changes.
Regardless of those changes, I think team review and approval would be easier if we have more information on alternative designs so the tradeoffs can be judged better. Hence the proposal for a research project: https://github.com/rust-lang/rust-project-goals/pull/155
How heavily will real-world usage depend on Cargo supporting binary dependencies Tracking Issue for RFC 3028: Allow "artifact dependencies" on bin, cdylib, and staticlib crates cargo#9096? Right now that is kind of a blocker to ergonomic rust shared libraries (at least for my use case)
We'd ideally need cargo to understand the concept of a dynamically linked dependency to make this an ergonomic feature. However, using (generated?) 'header-only' crates, we can make this ergonomic enough for a first usable version without any special cargo support.
How would this work with cargo install wouldn't cargo need to know to install both the binary and all its dynamically linked dependencies. Where would those dynamic libraries live and how would it work if one binary depends on foobar@1.* while another depends on foobar@2.* as I would expect both to result in a foobar.dll that are likely ABI incompatible.
Edit:
Also how should a dependency be handled that appears multiple times in a crates dependency graph as both dynamically and statically linked?
I think this should also support conditionally using stable ABIs of crate author's choice only when it's being compiled as dynamic library.
Say my project is statically linking to another crate that uses #[export] attributes a lot. Then many of its types would use different ABI which may not be optimal when I'm statically linking to it.
Compiler should compile those types using unstable default ABI for optimal performance.
This could be achieved by providing options like #[export(abi(C))] so that it can specify ABI itself instead of relying on extern or #[repr] which would conditionally apply other ABI when it's needed.
Or it could provide some cfg variable for crate's authors to use with #[cfg...] but I don't think that'd be compatible with extern
I think this should also support conditionally using stable ABIs of crate author's choice only when it's being compiled as dynamic library.
Say my project is statically linking to another crate that uses
#[export]attributes a lot. Then many of its types would use different ABI which may not be optimal when I'm statically linking to it. Compiler should compile those types using unstable default ABI for optimal performance.This could be achieved by providing options like
#[export(abi(C))]so that it can specify ABI itself instead of relying onexternor#[repr]which would conditionally apply other ABI when it's needed. Or it could provide some cfg variable for crate's authors to use with#[cfg...]but I don't think that'd be compatible withextern
nvm it's already handled in the rfc:
# Importing Exported Items
Normally, when using a crate as a dependency, any #[export] attributes of that crate have no effect and the dependency is statically linked into the resulting binary.
When explicitly specifying dynamic = true for the dependency with Cargo.toml, or when using a extern dyn crate …; statement in the source code, only the items marked as #[export] will be available and the dependency will be linked dynamically.