kafka2 plugin not works with SSL authentication method....error="Waiting for delivery timed out after 30 seconds"

[kunalv89]

trafficstars

Describe the bug

Team, Requesting your help to setup secure communication with SSL from Log aggregator (td-agent) to kafka broker, but getting error each time.

<match unity> @type kafka2 brokers 192.168.10.1:9092,192.168.10.2:9092 use_event_time true security_protocol ssl ssl_truststore_location "/etc/td-agent/keys/kafka.client.keystore.jks" ssl_truststore_password "*****" ssl_keystore_location "/etc/td-agent/keys/kafka.client.keystore.jks" ssl_keystore_password "*****" key_password "*****" max_send_retries 3 <format> @type json </format> <buffer> @type file path /var/log/td-agent/buffer/td flush_interval 3s </buffer> topic_key topicone default_topic topicone </match>

To Reproduce

Tried different-2 SSL parameters on @type kafka2 but didn't get works.

ssl_ca_cert ["/etc/td-agent/cert/CARoot.pem"] ssl_client_cert "/etc/td-agent/cert/client_cert.pem" ssl_client_cert_key "/etc/td-agent/cert/client_key.pem" ssl_client_cert_chain PEM

Expected behavior

Data should start transmitting from log aggregator to kafka broker. without SSL authentication i have tested successfully sends data to kafka broker.

Your Environment

- Fluentd version:1.11.5
- TD Agent version: 1.11.5
- Operating system: RHEL 7.9 (Maipo)
- Kernel version: 3.10.0-862.el7.x86_64

Your Configuration

<match unity> @type kafka2 brokers 192.168.10.1:9092,192.168.10.2:9092 use_event_time true security_protocol ssl ssl_truststore_location "/etc/td-agent/keys/kafka.client.keystore.jks" ssl_truststore_password "*****" ssl_keystore_location "/etc/td-agent/keys/kafka.client.keystore.jks" ssl_keystore_password "*****" key_password "*****" max_send_retries 3 <format> @type json </format> <buffer> @type file path /var/log/td-agent/buffer/td flush_interval 3s </buffer> topic_key topicone default_topic topicone </match>

Your Error Log

Error while passing ssl_ca_cert, ssl_client_cert, ssl_client_cert_key

2022-03-10 12:07:13 +0530 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2022-03-10 12:07:15 +0530 chunk="5d9a1a1ef31dea60431e4f4861a030e9" error_class=OpenSSL::SSL::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed"
2022-03-10 12:07:13 +0530 [warn]: #0 suppressed same stacktrace
2022-03-10 12:07:15 +0530 [warn]: #0 Send exception occurred: SSL_connect returned=1 errno=0 state=error: certificate verify failed
2022-03-10 12:07:15 +0530 [warn]: #0 Exception Backtrace : /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ruby-kafka-1.4.0/lib/kafka/ssl_socket_with_timeout.rb:69:in `connect_nonblock'
_______________________________________________________________________________________________________
Error while passing ssl_truststore_location, ssl_keystore_location

2022-03-11 10:35:03 +0530 [warn]: #0 suppressed same stacktrace
2022-03-11 10:35:05 +0530 [warn]: #0 Send exception occurred: Could not connect to any of the seed brokers:
- kafka://192.168.10.1:9092: Connection error EOFError: EOFError
- kafka://192.168.10.2_:9092: Connection error EOFError: EOFError
2022-03-11 10:35:05 +0530 [warn]: #0 Exception Backtrace : /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/ruby-kafka-1.4.0/lib/kafka/cluster.rb:454:in `fetch_cluster_info'

Additional context

No response

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days