Building cdylibs and plugins with cargo

[m-ou-se]

There's a few open issues related to building plugins for other software (as cdylibs) with cargo. With this issue I want to summarize them and try to work to a single solution.

The use case

Software like Python, PostgreSQL, and many others can dynamically load plugins. These plugins are dynamic libraries (i.e. cdylibs in Rust, aka .so/.dylib/.dll files) with certain symbols exposed. We're looking here at making such plugins using Rust.

The problems

1. It's unclear whether these should be [lib] or [[bin]] crates. [[bin]]s cannot be cdylibs, but a package can currently only contain one [lib]. These plugins will not be 'consumed' by other Rust crates as dependencies, so this limitation is unnecessary for this type of crate. #6351
2. A cdylib [lib] is not made available to integration tests to test. #8311 #7152 #8193
3. required-features = .. is ignored for [lib].
4. The output file name cannot be chosen. E.g. on Linux it will be lib{cratename}.so, even though it might need to be something like {cratename}.plugin, python-{cratename}.pyd, etc. https://github.com/rust-lang/cargo/issues/1706#issuecomment-442891843
5. Plugins might need to depend on symbols from the program they will be loaded in to. This requires -Clink-arg=-undefined=dynamic_lookup on Mac. It might also require not depending on a library that a regular bin target should depend on. #5881
6. It is assumed the [lib] crate will be linked into the [[bin]] crates: dependencies from build.rs are added to the [lib] crate but not the [[bin]] crates, to avoid duplicate symbols. Plugin crates should not be linked into [[bin]] crates. #7506
7. These type of crates should set enable Package::include_lockfile(). This is done for examples and binaries, but not for libraries.
8. Sometimes these files will also have an entry point, so you can run them as binary (like how GNU libc has an entry point that prints its version). Rustc doesn't make an entry point for cdylibs by default, and bin targets don't handle thread locals in a way that they can be loaded into other binaries. (Probably not a cargo issue.)
9. Maybe more?

Possible solutions

#6351 suggests allowing crate-type = ["cdylib"] for [[bin]]s. Although these type of crates share some properties with [[bin]] crates, there's also properties they do not share: Binaries should be installed with cargo install, but plugins should probably not (or at least not into /usr/bin). Binaries have their private items documented by default (as they have no public items), but plugin crates do have public items. Binaries can be run by cargo run, but these should not.

https://github.com/rust-lang/cargo/issues/6351#issuecomment-633682243 suggests adding a new [[clib]] instead. This would be a much bigger change to cargo, as the set of target types (lib, bins, examples, tests, benches) are used in a lot of places (including command line flags).

Questions

1. Is a 'plugin' fundamentally different from a binary? Should they be [[bin]]s, or do they need more special treatment?

2. If not [[bin]]s, are they fundamentally different from other cdylibs? Would a [[clib]] work for both plugins and other type of dynamic C libraries? Or does it need a separate [[cplugin]] or cplugin = true in Cargo.toml to be handled correctly?

3. Should 'cdylibs with an entry point' be supported? If so, as "libraries that also have an entry point" (e.g. [[clib]] also_bin = true), or as "binaries that are also loadable" (e.g. [[bin]] export_stuff = true)?

4. Probably more questions?

[m-ou-se]

Here's some of the behaviour I'd expect for these type of crates, compared to the current behaviour of lib and bin crates:

behaviour	[[bin]]	[lib]	Plugins	Other C libraries	comment
Installed by cargo install	:heavy_check_mark:	:x:*	:x:*	:x:*	Maybe? Probably a separate issue if yes.
Available in tests through CARGO_BIN_EXE_*	:heavy_check_mark:	:x:	:heavy_check_mark:*	:question:*	CARGO_LIB_* or something
Used in fingerprints	:x:	:heavy_check_mark:	:x:	:x:
cargo doc documents private items by default	:heavy_check_mark:	:x:	:x:	:x:
Added as dependency to other binaries (and tests, etc.)	:x:	:heavy_check_mark:	:x:	:x:
Might need -Clink-arg=-undefined=dynamic_lookup	:x:	:x:	:heavy_check_mark:	:question:
Run using cargo run	:heavy_check_mark:	:x:	:question:*	:x:	Using a runner for plugins can make sense
Triggers inclusion of lockfile	:heavy_check_mark:	:x:	:heavy_check_mark:	:heavy_check_mark:
Only built when requires-features available	:heavy_check_mark:	:x:	:heavy_check_mark:	:heavy_check_mark:
Depends on the [lib] target (if any)	:heavy_check_mark:	:x:	:heavy_check_mark:	:heavy_check_mark:
Built using cargo build --bins	:heavy_check_mark:	:x:	:x:*	:x:*	Maybe?
Custom output filename is useful	:question:	:x:	:heavy_check_mark:	:question:
Multiple allowed in one package	:heavy_check_mark:	:x:	:heavy_check_mark:	:heavy_check_mark:
Can be staticlib	:x:	:heavy_check_mark:*	:x:	:heavy_check_mark:	Might not make sense for a regular [lib], as Rust code can't use it as a dependency.
(Can) have an entry point	:heavy_check_mark:	:x:	:question:	:question:
localexec TLS model	:heavy_check_mark:	:x:	:x:	:x:	(depends on --crate-type given to rustc)

The last two columns are very similar. So this suggests that the answer to question 1 and 2 would be that a new [[clib]] type could suffice for both plugins and other cdylib crates.

[m-ou-se]

After thinking about this a bit more, I think the best way forward is indeed a new section in the toml, something like [[clib]]. This would cover both regular (C) libraries and plugins. The table shows that these two columns are almost entirely identical, but different enough from [lib] and [[bin]] to warrant a new section.

Other than some bikeshedding about the exact name (clib? Does it need C in the name, or is there a better name?), I think the expected behaviour of this new addition is be clear from the table above.

So I propose adding [[clib]] (name to be bikeshed later), with the following properties:

• Multiple [[clib]]s allowed in one package.
• Made available to integration tests using CARGO_LIB_<name> (just like CARGO_BIN_EXE_<name>).
• Can be gated behind requires-features.
• Not part of cargo build --bins. (Maybe a new flag would be useful.)
• Allows a custom file name for the output. (So doesn't force e.g. lib<name>.so.)
• Allows the same crate-types as a [lib].
• Not used for fingerprints.
• Compiled with the [lib] target as a dependency (if available).

Special linker flags (e.g. -undefined=dynamic_lookup), runners (cargo run --clib=..?), and cdylibs with an entry point are the three things I think are better to not solve right now. Once [[clib]] is a thing, those features can be discussed and added later.

I can work on implementing this, but would first like some confirmation from the Cargo team that this change would be acceptable/wanted.

[jeff-davis]

@m-ou-se Thank you for working on this. What you describe satisfies my use case almost perfectly.

One other thing to consider is what happens with cargo test. A plugin-style shared cdylib will fail immediately due to unresolved symbols because it can't link the test binary. We should at least fail immediately with a clear error message; or perhaps run an integration test instead.

cargo install would be useful if it could actually install it in the right place (i.e. run some custom code that figures out where to install it). Otherwise, we'd have to create a [[bin]] that can do the installation, but that could be confusing because cargo install would then install the installation binary.

[dherman]

Thank you for this writeup, @m-ou-se! I work on Neon, which uses Rust to implement Node.js plugins. I think we fit in your category as well, and right now we're a bit blocked specifically on your problem no. 4, i.e., the ability to rename the output file (Node expects it to be called *.node).

[guilhermewerner]

This is a great feature, and it covers a lot of use cases. Some i can think of now:

1. Creation of plugins to be consumed by other languages/applications.

As already said, the ability to use a library made with rust in other environments is very welcome, I've seen several similar issues, and I'm interested in it myself.

One problem I encountered was making a plugin system, where a library exports an extern "C" function to get a pointer to the plugin. This works perfectly with a cdylib, through dynamic loading, however if the plugin is statically linked as an rlib, the function name collides when linking, as noted:

• https://github.com/rust-lang/rust/issues/20267#issuecomment-899044709
• https://github.com/eclipse-zenoh/zenoh/issues/89#issuecomment-831241728

That way we could concentrate the plugin code, being able to have an api usable as a normal rust lib, and allowing external use through the dynamic library.

2. The possibility of having more than one cdylib per package.

More or less related to the previous topic. This possibility would be interesting, for reusing the code of the main lib, and having variations in multiple cdylibs, without the need to create another package.

An example of these variations would be how it is done in Unreal Engine, where it is possible to have more than one "module" per project. These modules are nothing less than a dynamic library.

https://docs.unrealengine.com/4.26/en-US/ProgrammingAndScripting/GameplayArchitecture/Gameplay/#:~:text=In%20the%20same,proprietary%20package%20files.

If I think of any more pertinent cases I'll add them here later. And it would be nice to know the opinion of more people, each person has different experiences and can come up with amazing ideas 😀

[guilhermewerner]

While this functionality doesn't go forward, I'm using an [[example]], as it's the only way to create multiple cdylibs. It's not ideal, but for now it works.

I've been looking into the cargo source code, to see if I could help.

[m-ou-se]

I just remembered I actually implemented this a while ago. I've just not been able to test any of it yet. (And there's probably a bunch of things to clean up and fix.)

I'll try to find some time to dig it up and send it as a PR to get this thing going.

[jmjoy]

Is there any follow-up to this issue?

[AlexDev404]

@m-ou-se https://github.com/rust-lang/cargo/issues/8628#issuecomment-1165227017

[JohnScience]

What's needed for moving this proposal forward? RFC and then implementation?

[nyurik]

There are currently 5 types of results a cargo build|check|clippy|... can work on: bins, benches, examples, lib, tests. This sometimes creates a confusion of what code will or won't be included, or surprising things like the fact that examples build [dep-dependencies]. On one hand, when there is 5, there is 6 - not a biggie. On the other, I am a bit hesitant about adding a new basic type to this already slightly confusing mix. The bins type keep resurfacing here, and it seems like this build type is rare enough to survive if it requires a few extra params rather than a dedicated type... i.e. in my old Windows days, DLL was considered a proper bin target... or i might be totally wrong. Here I tried to put together what a bin may look like.

[[bin]]
name = "mylib"
crate-type = "cdylib"
required-features = ["loadable_extension"]
document-private = true
run-harness = "..." # this one is tough, e.g. i need sqlite3 to use .load lib command?

Workaround

My current workaround is to use [[example]] section. This way I can create a cdylib binary, while keeping the core functionality in the [lib] that can be reused from other rust code. See sqlite-hashes and sqlite-compressions -- loadable SQLite extensions.