Failing to compile 0.28.0 for Windows and musl platforms with vendored openssl

[jszwedko]

Hey!

Apologies, I tried to nail this down myself, but have been banging my head against it all day and thought maybe you would have some insight.

We have a project (https://github.com/vectordotdev/vector) using rust-rdkafka that we are trying to upgrade from 0.27.0 to 0.28.0 to include https://github.com/fede1024/rust-rdkafka/commit/de421ba8f0afba6722a83b7bc455d669e9ce7b37 but our cross compilation for musl-based platfroms as well as native builds for Windows have errors linking a vendored openssl that they didn't for 0.27.0.

For the cross compilation to musl this looks like:

 = note: /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_set_certs':
          /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1348: undefined reference to `ENGINE_load_ssl_client_cert'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_ctx_term':
          /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1421: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_ctx_init_engine':
          /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1442: undefined reference to `ENGINE_by_id'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1444: undefined reference to `ENGINE_by_id'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1453: undefined reference to `ENGINE_ctrl_cmd_string'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1456: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1463: undefined reference to `ENGINE_ctrl_cmd_string'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1465: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1472: undefined reference to `ENGINE_ctrl_cmd_string'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1473: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1480: undefined reference to `ENGINE_init'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1481: undefined reference to `ENGINE_free'
          /usr/local/bin/../lib/gcc/aarch64-linux-musl/9.2.0/../../../../aarch64-linux-musl/bin/ld: /target/aarch64-unknown-linux-musl/debug/deps/librdkafka_sys-315d339ecd8675f4.rlib(rdkafka_ssl.c.o): in function `rd_kafka_ssl_ctx_init':
          /cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-4.2.0+1.8.2/librdkafka/src/rdkafka_ssl.c:1627: undefined reference to `ENGINE_free'
          collect2: error: ld returned 1 exit status
          
  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

For Windows this looks like: https://gist.github.com/jszwedko/5e959ee2cc4c1508238ad8a146d20192

They seem plausibly related.

The features we use are:

rdkafka v0.28.0 cmake_build,libz,ssl,tokio,zstd

For openssl:

openssl v0.10.38 vendored

You can see the CI build failures here: https://github.com/vectordotdev/vector/runs/5029404180?check_suite_focus=true

I put together an simplified example repo that seems to reproduce it for me using just rdkafka with the ssl/vendored feature here: https://github.com/jszwedko/rdkafka-vendored-ssl

In there I run:

cargo build  --target x86_64-unknown-linux-musl

Any insight would be appreciated 🙂

[benesch]

Hm, nothing changed about the build system between v0.27.0 and v0.28.0 that I can see, so really not sure what's going wrong. I wonder if it's librdkafka itself that's misbehaving. You could try adjusting vectordotdev/vector#10302 to update rdkafka without updating rdkafka-sys. That would help to isolate the problem, and would let you pick up the bugfix without bumping librdkafka itself.

Otherwise my reply in https://github.com/fede1024/rust-rdkafka/issues/447#issuecomment-1047425549 applies here too. If you can re-run with cargo build -vv I can take a quick skim, but I'm afraid my time for debugging cross-compiling issues is basically zero.

[jszwedko]

Thanks for taking a peek @benesch ! I reran a clean build of my dummy project with cargo build -vv. The log is here:

https://gist.github.com/d52bf10d2ea38df0658a79f4bd90c5f7

I'm curious if anything jumps out to you.

[jszwedko]

I attempted to just upgrade rdkafka to 0.28.0 and keep rdkafka-sys at 4.0.0, but ran into the same failure.

[0x003e]

@jszwedko try this: https://github.com/fede1024/rust-rdkafka/issues/479#issuecomment-1159406724

[neuronull]

@jszwedko try this: #479 (comment)

Hi all, I'm investigating this issue with @jszwedko .

@0x003e , I tried that solution and while it did not work in moving from 0.27 to 0.28 , it did help reveal something intriguing.

If I specify the following, I get the same error as in the original comment (undefined reference to ENGINE_*).

rdkafka = { git="https://github.com/fede1024/rust-rdkafka", rev="v0.27.0", default-features = false, features = ["ssl-vendored", "libz", "tokio", "zstd"] }

However, if I override the rdkafka-sys to the version in v0.26.0 , compilation succeeds.

[patch.'https://github.com/fede1024/rust-rdkafka']
rdkafka-sys = { path = '../rust-rdkafka-old/rdkafka-sys' }

(where rust-rdkafka-old is a checkout of v0.26.0)

I believe @0x003e is on to something with this comment: https://github.com/fede1024/rust-rdkafka/issues/479#issuecomment-1159447281

I also saw here: https://github.com/edenhill/librdkafka/releases/tag/v1.7.0 , that one of the features of v1.7.0 of librdkafka, was to include support for OpenSSL Engine... (which is what all of the linker errors are centered on).

Wondering if the openssl-sys version that rdkafka-sys specifies (0.9.48) needs to be upgraded, to a version that includes OpenSSL Engine support. It appears the openssl-sys version in rdkafka-sys has not been modified since 2019.

[neuronull]

On further inspection, I found that it is not the SSL version that is the problem. The problem is that whenever cross-compiling for musl , the openssl-sys is setting the no-engine flag when configuring the openssl for compilation.

This combined with the fact that librdkafka v1.7.0 introduced a dependency on the SSL Engine feature, means that the cross compilation won't work without a modification somewhere.

It seems either openssl-sys needs to allow SSL Engine feature for musl cross-compilation, or rust-rdkafka / librdkafka needs to make SSL Engine dependency configurable.

This is where the openssl-sys is setting the no-engine flag:

https://github.com/alexcrichton/openssl-src-rs/blob/0e9ec2f8b30ee89a8d1a486e100e03718a960665/src/lib.rs#L197-L200

[trinity-1686a]

it's now possible to force-enable engine on musl thanks to alexcrichton/openssl-src-rs#151

[build-dependencies]
openssl-src = { version = "111.24.0" , features = ["force-engine"] }

On alpine it requires to install linux-headers (otherwise it indeed fails to compile due to missing linux/version.h). Some part of the engine module depends on linux >= 4.1.0 (any version not EOL is fine), checked both at compile time and runtime by openssl