fluentd-kubernetes-daemonset
fluentd-kubernetes-daemonset copied to clipboard
Published
20 hours ago •
fluent
fluent
debian-kafka2 after v1.16.3-debian-kafka2-2.0 and above fails to start kafka output plugin
Describe the bug
All tags after the version of v1.16.3-debian-kafka2-2.0, i.e. v1.16.3-debian-kafka2-2.0, v1.16.3-debian-kafka2-2.1 and v1.16.5-debian-kafka2-1.0, suffer from this issue. They are unable to correctly start the rdkafka output plugin for fluentd.
In version fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.0 when the container is started with the configuration mentione in the 'To reproduce' section the below errors can be seen:
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:226:in `block (2 levels) in native_config'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:216:in `each'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:216:in `block in native_config'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 <internal:kernel>:90:in `tap'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:214:in `native_config'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:178:in `producer'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:387:in `block in get_producer'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:384:in `synchronize'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:384:in `get_producer'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:421:in `write'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1225:in `try_flush'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #2 Send exception occurred: Configuration property "ssl.ca.location" not supported in this build: OpenSSL not available at build time at /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:226:in `block (2 levels) in native_config'
A bit of checking shows that the libssl-dev is not available any more in the container. This is the result of from fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-1.0
# apt list | grep ssl
libssl-dev/now 1.1.1w-0+deb11u1 amd64 [installed,local]
libssl1.1/now 1.1.1w-0+deb11u1 amd64 [installed,local]
openssl/now 1.1.1w-0+deb11u1 amd64 [installed,local]
where everything seems to work as expected.
The result from the fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.0 is
# apt list | grep ssl
libssl3/now 3.0.11-1~deb12u2 amd64 [installed,local]
openssl/now 3.0.11-1~deb12u2 amd64 [installed,local]
You can see that libssl-dev is missing.
On fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.1 the error changes and becomes:
fluentd-1 | /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/dynamic_library.rb:65:in `load_library': Could not open library '/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/../../ext/librdkafka.so': libsasl2.so.2: cannot open shared object file: No such file or directory. (LoadError)
fluentd-1 | Searched in <system library path>, /usr/lib, /usr/local/lib, /opt/local/lib
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:95:in `block in ffi_lib'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `map'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `ffi_lib'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:18:in `<module:Bindings>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:7:in `<module:Rdkafka>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:5:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka.rb:9:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:159:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:159:in `rescue in require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:39:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:6:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:149:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'
fluentd-1 | <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- rdkafka (LoadError)
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:6:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:149:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'
Looking into the container we can see that libsasl2 is not present:
# ldd $(gem contents rdkafka | grep librdkafka.so)
linux-vdso.so.1 (0x00007ffe842b7000)
/usr/lib/libjemalloc.so.2 (0x00007ffad7d8b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffad7caa000)
libsasl2.so.2 => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffad7ac9000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffad8226000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffad78af000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffad788d000)
The same happens with the latest tag you provide as of today: fluent/fluentd-kubernetes-daemonset:v1.16.5-debian-kafka2-1.0 Keep in mind that in both these last image the openssl-dev is not present, either.
To me it looks like this commit created this issue by purgin still necessary libraries: https://github.com/fluent/fluentd-kubernetes-daemonset/blob/29fdf0324742de37b635e1fa3884366e4f38b183/templates/Dockerfile.erb#L52
To Reproduce
Use below configuration to start the fluentd container:
`
Container using the above image tags, will create errors long before it tries to send data to the broker. The errors are described above per image tag.
Expected behavior
No errors are raised during the plugin's initialization.
Your Environment
- Tag of using fluentd-kubernetes-daemonset:
fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.0
fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.1
fluent/fluentd-kubernetes-daemonset:v1.16.5-debian-kafka2-1.0
Your Configuration
<source>
@type forward
port 29999
bind 0.0.0.0
</source>
<match>
@type rdkafka2
brokers "localhost:9092"
default_topic "test_topic"
ssl_ca_cert "/certs/test.crt"
<format>
@type json
</format>
</match>
Your Error Log
fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.0
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 failed to flush the buffer. retry_times=3 next_retry_time=2024-04-08 13:54:16 +0000 chunk="615961fb1f6b351762de7dee1a9625a7" error_class=Rdkafka::Config::ConfigError error="Configuration property \"ssl.ca.location\" not supported in this build: OpenSSL not available at build time"
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:226:in `block (2 levels) in native_config'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:216:in `each'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:216:in `block in native_config'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 <internal:kernel>:90:in `tap'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:214:in `native_config'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:178:in `producer'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:387:in `block in get_producer'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:384:in `synchronize'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:384:in `get_producer'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:421:in `write'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1225:in `try_flush'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #1 /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
fluentd-1 | 2024-04-08 13:54:08 +0000 [warn]: #2 Send exception occurred: Configuration property "ssl.ca.location" not supported in this build: OpenSSL not available at build time at /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/config.rb:226:in `block (2 levels) in native_config'
fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-kafka2-2.1
fluentd-1 | /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/dynamic_library.rb:65:in `load_library': Could not open library '/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/../../ext/librdkafka.so': libsasl2.so.2: cannot open shared object file: No such file or directory. (LoadError)
fluentd-1 | Searched in <system library path>, /usr/lib, /usr/local/lib, /opt/local/lib
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:95:in `block in ffi_lib'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `map'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `ffi_lib'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:18:in `<module:Bindings>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:7:in `<module:Rdkafka>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:5:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka.rb:9:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:159:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:159:in `rescue in require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:39:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:6:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:149:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'
fluentd-1 | <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- rdkafka (LoadError)
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:6:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:103:in `block in search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:100:in `search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/registry.rb:44:in `lookup'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:169:in `new_impl'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin.rb:114:in `new_output'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:130:in `add_match'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:74:in `block in configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/agent.rb:64:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/root_agent.rb:149:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:105:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/engine.rb:80:in `run_configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/supervisor.rb:571:in `run_supervisor'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.3/bin/fluentd:15:in `<top (required)>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'
fluent/fluentd-kubernetes-daemonset:v1.16.5-debian-kafka2-1.0
fluentd-1 | /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/dynamic_library.rb:65:in `load_library': Could not open library '/fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/../../ext/librdkafka.so': libsasl2.so.2: cannot open shared object file: No such file or directory. (LoadError)
fluentd-1 | Searched in <system library path>, /usr/lib, /usr/local/lib, /opt/local/lib
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:95:in `block in ffi_lib'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `map'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `ffi_lib'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:18:in `<module:Bindings>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:7:in `<module:Rdkafka>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka/bindings.rb:5:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/rdkafka-0.11.1/lib/rdkafka.rb:9:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:40:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:6:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:103:in `block in search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:100:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:100:in `search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:44:in `lookup'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin.rb:169:in `new_impl'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin.rb:114:in `new_output'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:130:in `add_match'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:74:in `block in configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:64:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:64:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/root_agent.rb:149:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/engine.rb:105:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/engine.rb:80:in `run_configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/supervisor.rb:571:in `run_supervisor'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/bin/fluentd:15:in `<top (required)>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'
fluentd-1 | <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': cannot load such file -- rdkafka (LoadError)
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb:6:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:103:in `block in search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:100:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:100:in `search'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/registry.rb:44:in `lookup'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin.rb:169:in `new_impl'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/plugin.rb:114:in `new_output'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:130:in `add_match'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:74:in `block in configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:64:in `each'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/agent.rb:64:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/root_agent.rb:149:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/engine.rb:105:in `configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/engine.rb:80:in `run_configure'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/supervisor.rb:571:in `run_supervisor'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/lib/fluent/command/fluentd.rb:352:in `<top (required)>'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from <internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/gems/fluentd-1.16.5/bin/fluentd:15:in `<top (required)>'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `load'
fluentd-1 | from /fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd:25:in `<main>'
Additional context
No response
