fluent-plugin-kafka icon indicating copy to clipboard operation
fluent-plugin-kafka copied to clipboard

Published 20 hours ago • verified publisher icon fluent

fluent-input-kafka use ssl error: SSL_CTX_use_certificate: ca md too weak

Open ly123-liu opened this issue 2 years ago • 2 comments

Describe the bug

we use fluent to consume kafka messages with ssl cert , fluent start with error below:

2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak" 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in initialize' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in new' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in initialize' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in new' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in open' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:101:in block in send_request' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/instrumenter.rb:23:in instrument' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:100:in send_request' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:200:in send_request' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:44:in fetch_metadata' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:427:in block in fetch_cluster_info' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in each' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in fetch_cluster_info' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:402:in cluster_info' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:102:in refresh_metadata!' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:106:in refresh_metadata_if_necessary!' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:452:in random_broker' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:382:in list_topics' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:634:in cluster_topics' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:614:in subscribe_to_regex' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:606:in block in scan_for_subscribing' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in each' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in scan_for_subscribing' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:118:in subscribe' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:229:in block in setup_consumer' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in each' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in setup_consumer' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:202:in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/compat/call_super_mixin.rb:42:in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:200:in block in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:189:in block (2 levels) in lifecycle' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in each' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in block in lifecycle' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in each' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in lifecycle' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:199:in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:248:in start' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:147:in run' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:590:in block in run_worker' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:825:in main_process' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:584:in run_worker' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/command/fluentd.rb:338:in <top (required)>' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in require' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in require' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/bin/fluentd:8:in <top (required)>' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in load' 2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `

' 2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak" 2022-01-12 15:00:45 +0800 [error]: #0 suppressed same stacktrace

To Reproduce

since we generate kafka cert without set message digest, default message digest seems sha1WithRSAEncryption , is there fluent-kafka-input plugin can check tls with sha1 and continue to work ?

    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN = Yingmi Infra Kafka RootCA
    Validity
        Not Before: Mar  8 02:15:36 2021 GMT
        Not After : Feb 12 02:15:36 2121 GMT
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption

Expected behavior

1

Your Environment

no special

Your Configuration

@type kafka_group

brokers kafka.infra:9092 consumer_group log-fluentd-test topics /log-prod.*/ format text message_key <key (Optional, for text format only, default is message)> #kafka_message_key <key (Optional, If specified, set kafka's message key to this key)> #add_headers <If true, add kafka's message headers to record> #add_prefix <tag prefix (Optional)> #add_suffix <tag suffix (Optional)> retry_emit_limit 3 time_source record time_format <string (Optional when use_record_time is used)>

ruby-kafka consumer options

max_bytes 1048576 #max_wait_time (integer) :default => nil (Use default of ruby-kafka) #min_bytes (integer) :default => nil (Use default of ruby-kafka) #offset_commit_interval (integer) :default => nil (Use default of ruby-kafka) #offset_commit_threshold (integer) :default => nil (Use default of ruby-kafka) #fetcher_max_queue_size (integer) :default => nil (Use default of ruby-kafka) start_from_beginning true

get_kafka_client_log true connect_timeout 10 socket_timeout 30 ssl_verify_hostname false ssl_ca_cert /ca.crt ssl_client_cert /log.crt ssl_client_cert_key /log.key ssl_client_cert_chain PEM

Your Error Log

2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak"
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in `initialize'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in `new'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/ssl_socket_with_timeout.rb:59:in `initialize'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in `new'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:130:in `open'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:101:in `block in send_request'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/instrumenter.rb:23:in `instrument'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/connection.rb:100:in `send_request'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:200:in `send_request'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/broker.rb:44:in `fetch_metadata'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:427:in `block in fetch_cluster_info'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:422:in `fetch_cluster_info'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:402:in `cluster_info'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:102:in `refresh_metadata!'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:106:in `refresh_metadata_if_necessary!'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:452:in `random_broker'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:382:in `list_topics'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:634:in `cluster_topics'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:614:in `subscribe_to_regex'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:606:in `block in scan_for_subscribing'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:601:in `scan_for_subscribing'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/ruby-kafka-1.3.0/lib/kafka/consumer.rb:118:in `subscribe'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:229:in `block in setup_consumer'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:221:in `setup_consumer'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/in_kafka_group.rb:202:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:200:in `block in start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:188:in `block in lifecycle'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in `each'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:175:in `lifecycle'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/root_agent.rb:199:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:248:in `start'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/engine.rb:147:in `run'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:590:in `block in run_worker'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:825:in `main_process'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/supervisor.rb:584:in `run_worker'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/lib/fluent/command/fluentd.rb:338:in `<top (required)>'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/gems/fluentd-1.9.1/bin/fluentd:8:in `<top (required)>'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'
  2022-01-12 15:00:45 +0800 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
2022-01-12 15:00:45 +0800 [error]: #0 unexpected error error_class=OpenSSL::SSL::SSLError error="SSL_CTX_use_certificate: ca md too weak"
  2022-01-12 15:00:45 +0800 [error]: #0 suppressed same stacktrace

Additional context

No response

ly123-liu avatar Jan 12 '22 07:01 ly123-liu

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] avatar Apr 12 '22 10:04 github-actions[bot]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] avatar Jul 27 '22 10:07 github-actions[bot]

This issue was automatically closed because of stale in 30 days

github-actions[bot] avatar Aug 26 '22 10:08 github-actions[bot]