fluent-bit
fluent-bit copied to clipboard
fluent
Fluent-Bit with FIPS and Kafka output: SSL routines::library has no ciphers
Bug Report
Describe the bug We built Fluent Bit against OpenSSL 3.4.0 with FIPS enabled and configured only FIPS provider. It looks like that Fluent-Bit or Kafka output plugin doesn't support FIPS algorithms and they try to load the legacy. Can we do something with that?
[2024/11/28 10:18:28] [ info] [fluent bit] version=3.2.0, commit=76910b2cc7, pid=1
...
[2024/11/28 10:18:28] [error] [openssl] could not create context
[2024/11/28 10:18:28] [error] [tls] could not create TLS backend
...
[2024/11/28 10:18:28] [error] [output:kafka:kafka_app_logs] fluent-bit#producer-1: [thrd:app]: error:0A0000A1:SSL routines::library has no ciphers
[2024/11/28 10:18:28] [error] [output:kafka:kafka_app_logs] fluent-bit#producer-1: [thrd:app]: error:0A0000A1:SSL routines::library has no ciphers
[2024/11/28 10:18:28] [error] [output:kafka:kafka_app_logs] failed to create producer: SSL_CTX_new() failed: error:0A0000A1:SSL routines::library has no ciphers
Expected behavior OpenSSL context is created with FIPS enabled.
Your Environment Kubernetes and Ubuntu 22.04.
Additional context We need to enable FIPS to be FedRAMP compliant.
We're going to need a lot more details on how you built and how you're running.
I pushed my changes here. It looks like working Kafka is not needed to reproduce the problem. Simple fluent-bit instance with Kafka output and SSL configured is more than enough.
fluent-bit-1 | [2024/12/03 22:48:17] [2024/12/03 22:48:17] [error] [output:kafka:kafka.0] failed to create producer: SSL_CTX_new() failed: ssl/ssl_lib.c:3977:SSL_CTX_new_ex error:0A0000A1:SSL routines::library has no ciphers
fluent-bit-1 | [engine] caught signal (SIGSEGV)
fluent-bit-1 | #0 0xaaaac8539a87 in flb_out_kafka_destroy() at plugins/out_kafka/kafka_config.c:232
fluent-bit-1 | #1 0xaaaac8539f73 in flb_out_kafka_create() at plugins/out_kafka/kafka_config.c:173
fluent-bit-1 | #2 0xaaaac8538277 in cb_kafka_init() at plugins/out_kafka/kafka.c:77
fluent-bit-1 | #3 0xaaaac844bc57 in flb_output_init_all() at src/flb_output.c:1314
fluent-bit-1 | #4 0xaaaac845a5df in flb_engine_start() at src/flb_engine.c:841
fluent-bit-1 | #5 0xaaaac8439ce3 in flb_lib_worker() at src/flb_lib.c:763
fluent-bit-1 | #6 0xffff84e7ee8f in start_thread() at reate.c:442
fluent-bit-1 | #7 0xffff84ee7b1b in thread_start() at sysv/linux/aarch64/clone.S:79
fluent-bit-1 | #8 0xffffffffffffffff in ???() at ???:0
fluent-bit-1 exited with code 133
To run the example just run docker-compose up or run ./run-example.sh.
Please let me know if you need anything else. We're really looking for some advice here. Thanks.
@maxio89 We've been building Fluent Bit on Ubuntu 22 with OpenSSL FIPS 3.3 for a while and it's been working fine, we've recently switched to OpenSSL FIPS 3.4.0 and it's failing.
Out dockerfile, almost 1:1 to https://github.com/fluent/fluent-bit/blob/master/packaging/distros/ubuntu/Dockerfile but just targetting one OS (Ubuntu 22), fails at the bash -c make step.
[100%] Linking C executable ../bin/fluent-bit
/usr/bin/ld: ../library/libfluent-bit.a(flb_hash.c.o): in function `flb_hash_init':
/tmp/fluent-bit/src/flb_hash.c:72: undefined reference to `EVP_MD_CTX_get_size_ex'
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/fluent-bit-bin.dir/build.make:416: bin/fluent-bit] Error 1
make[1]: *** [CMakeFiles/Makefile2:8608: src/CMakeFiles/fluent-bit-bin.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
This is the same for a currently working build of FB 3.1.8 we know works on OpenSSL 3.3, and a new FB 3.2.5 build, also fails. Both work though on OpenSSL 3.3
We have the same versions of all software between the different builds
cmake --version
cmake version 3.22.1
g++ --version
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
ld --version
GNU ld (GNU Binutils for Ubuntu) 2.38
Were you able to get anything working with OpenSSL 3.4?
This was also in the changelog for OpenSSL 3.3 to 3.4
XOF Digest API improvements
EVP_MD_CTX_get_size() and EVP_MD_CTX_size are macros that were aliased to EVP_MD_get_size which returns a constant value. XOF Digests such as SHAKE have an output size that is not fixed, so calling EVP_MD_get_size() is not sufficent. The existing macros now point to the new function EVP_MD_CTX_get_size_ex() which will retrieve the "size" for a XOF digest, otherwise it falls back to calling EVP_MD_get_size(). Note that the SHAKE implementation did not have a context getter previously, so the "size" will only be able to be retrieved with new providers.
According to that, in https://github.com/fluent/fluent-bit/blob/master/src/flb_hash.c#L72, where it references
context->digest_size = EVP_MD_CTX_size(context->backend_context);
EVP_MD_CTX_size should point to EVP_MD_CTX_get_size_ex, but it's getting an undefined reference error.
/tmp/fluent-bit/src/flb_hash.c:72: undefined reference to `EVP_MD_CTX_get_size_ex'
My issue above was resolved by forcing the correct libraries to be used, overriding the system ones.
https://github.com/openssl/openssl/issues/26621#issuecomment-2632639510
Adding this to my dockerfile got it working, for the updated path to the local install of OpenSSL and the path of our .pc pkg-config files
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig
ENV OPENSSL_ROOT_DIR=/usr/local
@alex-rowe can you submit a PR to update the containers here if that resolves it too?
@alex-rowe can you submit a PR to update the containers here if that resolves it too?
@patrick-stephens this would be specific to our local builds I think, as it's based on a non-default version of OpenSSL we're compiling.
If you're just using the default provided OpenSSL that comes with Ubuntu and installing the packages like in the packaging dockerfiles, there shouldn't be an issue.
OK, maybe send a docs PR then for the building side: https://github.com/fluent/fluent-bit-docs/blob/master/installation/sources/build-and-install.md
For now I'll close the issue then as seems to be a specific build config required.