Static libraries link error for rustbitcoin and rustminiscript modules

[luisfg30]

When building both the modules rustbitcoin and rustminiscript there is a linker error due to them using the lib rustsecp256k1 as an internal dependency. Because both modules generate a static library used by the main program it generates a name conflict at the link stage.

Complete output of the make command on the root folder:

clang++  -DRUST_BITCOIN -DRUST_MINISCRIPT -DBITCOIN_CORE -DBITCOIN_CORE -fsanitize=address,fuzzer -Wall -Wextra -std=c++20 -I include -I .  main.cpp modules/bitcoin/module.a modules/btcd/module.a modules/rustbitcoin/module.a modules/rustminiscript/module.a driver.o include/bitcoinfuzz/basemodule.o -o bitcoinfuzz
/usr/bin/ld: modules/rustminiscript/module.a(secp256k1_sys-57793bebcb29e281.secp256k1_sys.e72ffb579ffdca46-cgu.0.rcgu.o): in function `rustsecp256k1_v0_10_0_context_create':
secp256k1_sys.e72ffb579ffdca46-cgu.0:(.text.rustsecp256k1_v0_10_0_context_create+0x0): multiple definition of `rustsecp256k1_v0_10_0_context_create'; modules/rustbitcoin/module.a(secp256k1_sys-5644b611ed0ca1e2.secp256k1_sys.d9c00a96ab6dd27c-cgu.0.rcgu.o):secp256k1_sys.d9c00a96ab6dd27c-cgu.0:(.text.rustsecp256k1_v0_10_0_context_create+0x0): first defined here
/usr/bin/ld: modules/rustminiscript/module.a(secp256k1_sys-57793bebcb29e281.secp256k1_sys.e72ffb579ffdca46-cgu.0.rcgu.o): in function `rustsecp256k1_v0_10_0_default_illegal_callback_fn':
secp256k1_sys.e72ffb579ffdca46-cgu.0:(.text.rustsecp256k1_v0_10_0_default_illegal_callback_fn+0x0): multiple definition of `rustsecp256k1_v0_10_0_default_illegal_callback_fn'; modules/rustbitcoin/module.a(secp256k1_sys-5644b611ed0ca1e2.secp256k1_sys.d9c00a96ab6dd27c-cgu.0.rcgu.o):secp256k1_sys.d9c00a96ab6dd27c-cgu.0:(.text.rustsecp256k1_v0_10_0_default_illegal_callback_fn+0x0): first defined here
/usr/bin/ld: modules/rustminiscript/module.a(secp256k1_sys-57793bebcb29e281.secp256k1_sys.e72ffb579ffdca46-cgu.0.rcgu.o): in function `rustsecp256k1_v0_10_0_default_error_callback_fn':
secp256k1_sys.e72ffb579ffdca46-cgu.0:(.text.rustsecp256k1_v0_10_0_default_error_callback_fn+0x0): multiple definition of `rustsecp256k1_v0_10_0_default_error_callback_fn'; modules/rustbitcoin/module.a(secp256k1_sys-5644b611ed0ca1e2.secp256k1_sys.d9c00a96ab6dd27c-cgu.0.rcgu.o):secp256k1_sys.d9c00a96ab6dd27c-cgu.0:(.text.rustsecp256k1_v0_10_0_default_error_callback_fn+0x0): first defined here
/usr/bin/ld: modules/rustminiscript/module.a(secp256k1_sys-57793bebcb29e281.secp256k1_sys.e72ffb579ffdca46-cgu.0.rcgu.o): in function `secp256k1_sys::secp256k1_context_destroy':
secp256k1_sys.e72ffb579ffdca46-cgu.0:(.text._ZN13secp256k1_sys25secp256k1_context_destroy17h0fe24367d38b07ebE+0x0): multiple definition of `rustsecp256k1_v0_10_0_context_destroy'; modules/rustbitcoin/module.a(secp256k1_sys-5644b611ed0ca1e2.secp256k1_sys.d9c00a96ab6dd27c-cgu.0.rcgu.o):secp256k1_sys.d9c00a96ab6dd27c-cgu.0:(.text._ZN13secp256k1_sys25secp256k1_context_destroy17hf63d5ce74b035c04E+0x0): first defined here
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:12: bitcoinfuzz] Error 1

[apoelstra]

Can you post the output of cargo tree? As you can see, these symbols are all renamed to have v0_10_0 in them -- the point of this renaming is that we should never have two versions of the library that have the same symbol.

Maybe we made a mistake with this -- where are you getting rust-secp 0.10.0 from? This was released in July 2018.

[brunoerg]

rust_bitcoin_lib v0.1.0 (/Users/brunogarcia/projects/bitcoinfuzz/modules/rustbitcoin/rust_bitcoin_lib)
└── bitcoin v0.32.5
    ├── base58ck v0.1.0
    │   ├── bitcoin-internals v0.3.0
    │   └── bitcoin_hashes v0.14.0
    │       ├── bitcoin-io v0.1.3
    │       └── hex-conservative v0.2.1
    │           └── arrayvec v0.7.6
    ├── bech32 v0.11.0
    ├── bitcoin-internals v0.3.0
    ├── bitcoin-io v0.1.3
    ├── bitcoin-units v0.1.2
    │   └── bitcoin-internals v0.3.0
    ├── bitcoin_hashes v0.14.0 (*)
    ├── hex-conservative v0.2.1 (*)
    ├── hex_lit v0.1.1
    └── secp256k1 v0.29.1
        ├── bitcoin_hashes v0.14.0 (*)
        └── secp256k1-sys v0.10.1
            [build-dependencies]
            └── cc v1.1.34
                └── shlex v1.3.0

rust_miniscript_lib v0.1.0 (/Users/brunogarcia/projects/bitcoinfuzz/modules/rustminiscript/rust_miniscript_lib)
└── miniscript v12.3.0
    ├── bech32 v0.11.0
    └── bitcoin v0.32.4
        ├── base58ck v0.1.0
        │   ├── bitcoin-internals v0.3.0
        │   └── bitcoin_hashes v0.14.0
        │       ├── bitcoin-io v0.1.3
        │       └── hex-conservative v0.2.1
        │           └── arrayvec v0.7.6
        ├── bech32 v0.11.0
        ├── bitcoin-internals v0.3.0
        ├── bitcoin-io v0.1.3
        ├── bitcoin-units v0.1.2
        │   └── bitcoin-internals v0.3.0
        ├── bitcoin_hashes v0.14.0 (*)
        ├── hex-conservative v0.2.1 (*)
        ├── hex_lit v0.1.1
        └── secp256k1 v0.29.1
            ├── bitcoin_hashes v0.14.0 (*)
            └── secp256k1-sys v0.10.1
                [build-dependencies]
                └── cc v1.2.1
                    └── shlex v1.3.0

[apoelstra]

Oh, derp, it's secp256k1-sys 0.10.1, which is pretty recent.

So it looks like your error is because you're trying to link rust-bitcoin and rust-miniscript together, which both have a copy of these C symbols....but I'm still not sure why this would cause a linker error. The two copies of the functions should be identical.

[apoelstra]

You can maybe try compiling one or both with the RUSTFLAGS described in https://github.com/rust-bitcoin/rust-secp256k1/tree/master/secp256k1-sys#linking-to-external-symbols

[luisfg30]

You can maybe try compiling one or both with the RUSTFLAGS described in https://github.com/rust-bitcoin/rust-secp256k1/tree/master/secp256k1-sys#linking-to-external-symbols

Yes. So far I was trying to build all modules off this project to understand how it works. Right now there is no need to build both rustbitcoin and rustminiscript together, but that might be usefull in the future. I will give a try with that flag --cfg=rust_secp_no_symbol_renaming .

But also I will search about alternative ways of building/linking the modules (perhaps dynamic linking the shared dependencies from modules)

[brunoerg]

Right now there is no need to build both rustbitcoin and rustminiscript together, but that might be usefull in the future.

Obviously there is no fuzz target that will need both of them, but build both together is useful, sometimes I may want to run a target that needs rust-bitcoin and sometimes I may want to run a target that needs rust-miniscript and I shouldn't have to compile everything again.

[erickcestari]

I think this highlights a good use case for building all components together. We should be able to run any fuzz target without repeatedly rebuilding bitcoinfuzz. This same linking issue also appeared in this CI run: https://github.com/brunoerg/bitcoinfuzz/actions/runs/13813187451/job/38639530776?pr=91#step:16:41

Being able to build both rust-bitcoin and rust-miniscript together would be beneficial, even though individual fuzz targets might only need one or the other. This approach would allow us to switch between targets that require different dependencies without having to recompile everything each time.

I will try adding --cfg=rust_secp_no_symbol_renaming to RUSTFLAGS variable and see it fixes it

[erickcestari]

Adding --cfg=rust_secp_no_symbol_renaming to RUSTFLAGS give undefined reference errors:

https://github.com/brunoerg/bitcoinfuzz/actions/runs/13813902574/job/38641980908pr=91#step:16:47

[luisfg30]

Adding --cfg=rust_secp_no_symbol_renaming to RUSTFLAGS give undefined reference errors:

https://github.com/brunoerg/bitcoinfuzz/actions/runs/13813902574/job/38641980908pr=91#step:16:47

Exactly, I tested it locally and got the same error

[moisesPompilio]

I've encountered similar Rust and Go(#93) errors when setting up Dockerfiles and docker-compose for the project. To solve this, I'm considering merging related modules: combining Go-based ones like lnd and btcd, and Rust-based ones like LDK, Rust bitcoin, and Rust Miniscript. This would simplify the build process by compiling them together instead of separately. However, I'm not sure if this is the best approach.

[luisfg30]

I've encountered similar Rust and Go(#93) errors when setting up Dockerfiles and docker-compose for the project. To solve this, I'm considering merging related modules: combining Go-based ones like lnd and btcd, and Rust-based ones like LDK, Rust bitcoin, and Rust Miniscript. This would simplify the build process by compiling them together instead of separately. However, I'm not sure if this is the best approach.

I am testing a strategy here. For rustbitcoin and rustminiscript modules both depends on the crate secp256k1 which is the one with the name conflicts. My ideia is to genereate another rust library that will work as a common dependency for the modules. This lib is just the FFI interface for secp256k1 and will be a dynamic lib (.so file). This way any number of rust modules that depends on int can link dynamically to it during the build process, and we do not need to combine all the rust modules in one big package. I am doing some code based on this: https://sbmueller.github.io/posts/rustcpp/ Soon I will post a PR here.

If it works you can use the same idea for the go modules

[apoelstra]

Isn't this library just secp256k1-sys?

[luisfg30]

Isn't this library just secp256k1-sys? Exactly, actually the repo explains a very similar use case.

But the two modules (rustbitcoin and rustminiscript) depends on this crate, they have this line in the Cargo.toml files secp256k1 = {version = "0.29.0", features = ["rand-std"]}. That will indirectly download an compile secp256k1-sys.

@apoelstra you suggest that we modify the code spinets from the modules to depend directly on secp256k1-sys instead of the secp256k1 crate?

[brunoerg]

I've encountered similar Rust and Go(#93) errors when setting up Dockerfiles and docker-compose for the project. To solve this, I'm considering merging related modules: combining Go-based ones like lnd and btcd, and Rust-based ones like LDK, Rust bitcoin, and Rust Miniscript. This would simplify the build process by compiling them together instead of separately. However, I'm not sure if this is the best approach.

I think this a bad idea. See: https://github.com/brunoerg/bitcoinfuzz/pull/98#pullrequestreview-2686481261

[apoelstra]

@luisfg30 no, I'm not suggesting anything. I'm just saying that whatever you were hoping to do by defining a "bindings-only crate", you can skip the step where you create the crate because it already exists.