fluent-plugin-mqtt-io
fluent-plugin-mqtt-io copied to clipboard
Published
20 hours ago •
toyokazu
toyokazu
How to reinitialize fluentd connection after getting SSL Handshake error?
I am using fluentd on Raspberry Pi 3 and internet connection is established with GSM 800c module. The connection is established and sends data over AWS IoT MQTT server. But sometimes there is SSL Handshake error and the fluentd keeps on showing SSL Handshake Error and it never gets resolved.
2019-02-11 12:28:10 +0000 [error]: #0 The other error occurs in out_mqtt.,OpenSS L::SSL::SSLError,SSL_connect returned=1 errno=0 state=unknown state: unknown ale rt type
2019-02-11 12:28:10 +0000 [error]: #0 Retry in 2 sec
2019-02-11 12:28:15 +0000 [error]: #0 Timeout error occurs in out_mqtt.,Timeout: :Error,execution expired
2019-02-11 12:28:15 +0000 [error]: #0 Retry in 1 sec
2019-02-11 12:28:15 +0000 [warn]: #0 failed to flush the buffer. retry_time=5 ne xt_retry_seconds=2019-02-11 12:28:31 +0000 chunk="5819d6f3d299ba0524777cb175e652 9b" error_class=OpenSSL::SSL::SSLError error="SSL_write: unknown alert type"
2019-02-11 12:28:15 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/openssl/buffering.rb: 322:in `syswrite'
2019-02-11 12:28:15 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/openssl/buffering.rb: 322:in `do_write'
2019-02-11 12:28:15 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/openssl/buffering.rb: 340:in `write'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:556:in `block in send_packet'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:555:in `synchronize'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:555:in `send_packet'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:328:in `disconnect'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/mqtt_proxy.rb:93:in `retry_connect'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/mqtt_proxy.rb:122:in `rescue in rescue_disconnecti on'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/mqtt_proxy.rb:111:in `rescue_disconnection'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/out_mqtt.rb:163:in `block in write'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/event.rb:323:in `each'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/event.rb:323:in `block in each'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/buffer/file_chunk.rb:172:in `open'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/event.rb:322:in `each'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/out_mqtt.rb:162:in `write'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/output.rb:1123:in `try_flush'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/output.rb:1423:in `flush_thread_run'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/output.rb:452:in `block (2 levels) in start'
2019-02-11 12:28:15 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-02-11 12:28:31 +0000 [error]: #0 The other error occurs in out_mqtt.,OpenSS L::SSL::SSLError,SSL_write: unknown alert type
2019-02-11 12:28:31 +0000 [error]: #0 Retry in 1 sec
2019-02-11 12:28:31 +0000 [warn]: #0 failed to flush the buffer. retry_time=6 ne xt_retry_seconds=2019-02-11 12:29:04 +0000 chunk="5819d6f3d299ba0524777cb175e652 9b" error_class=OpenSSL::SSL::SSLError error="SSL_write: unknown alert type"
2019-02-11 12:28:31 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/openssl/buffering.rb: 322:in `syswrite'
2019-02-11 12:28:31 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/openssl/buffering.rb: 322:in `do_write'
2019-02-11 12:28:31 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/openssl/buffering.rb: 340:in `write'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:556:in `block in send_packet'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:555:in `synchronize'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:555:in `send_packet'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/mqtt-0.5.0/lib/m qtt/client.rb:328:in `disconnect'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/mqtt_proxy.rb:93:in `retry_connect'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/mqtt_proxy.rb:128:in `rescue in rescue_disconnecti on'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/mqtt_proxy.rb:111:in `rescue_disconnection'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/out_mqtt.rb:163:in `block in write'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/event.rb:323:in `each'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/event.rb:323:in `block in each'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/buffer/file_chunk.rb:172:in `open'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/event.rb:322:in `each'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-mq tt-io-0.4.2/lib/fluent/plugin/out_mqtt.rb:162:in `write'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/output.rb:1123:in `try_flush'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/output.rb:1423:in `flush_thread_run'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin/output.rb:452:in `block (2 levels) in start'
2019-02-11 12:28:31 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.3.3/li b/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-02-11 12:29:04 +0000 [error]: #0 The other error occurs in out_mqtt.,OpenSS L::SSL::SSLError,SSL_write: unknown alert type
2019-02-11 12:29:04 +0000 [error]: #0 Retry in 1 sec
2019-02-11 12:29:04 +0000 [warn]: #0 failed to flush the buffer. retry_time=7 ne xt_retry_seconds=2019-02-11 12:30:01 +0000 chunk="5819d6f3d299ba0524777cb175e652 9b" error_class=OpenSSL::SSL::SSLError error="SSL_write: unknown alert type"
2019-02-11 12:29:04 +0000 [warn]: #0 suppressed same stacktrace
2019-02-11 12:30:01 +0000 [error]: #0 The other error occurs in out_mqtt.,OpenSSL::SSL::SSLError,SSL_write: unknown alert type
2019-02-11 12:30:01 +0000 [error]: #0 Retry in 1 sec
Is there any way that I can reinitialize fluentd connection after this error occurs? Thank You.
Hi,
Sorry for being late. Unfortunately, I don't have the same environment and I could not test it. I just tested AWS IoT with Wi-Fi connection and fluent-plugin-mqtt-io successfully reconnect to AWS IoT Broker in spite of frequent Wi-Fi OFF/ON. The following is a sample configuration for the test.
vi fluent-sample.conf
---
<source>
@type mqtt
host XXXXXXX.iot.ap-northeast-1.amazonaws.com
port 8883
topic test
client_id clientid2
<parse>
@type none
</parse>
<security>
use_tls true
<tls>
ca_file /path/to/aws/rootCA.crt
cert_file /path/to/aws/client.crt
key_file /path/to/aws/client.key
</tls>
</security>
@label @test
</source>
<label @test>
<match **>
@type mqtt
host XXXXXXXXX.iot.ap-northeast-1.amazonaws.com
port 8883
client_id clientid3
topic_rewrite_pattern '^([\w\/]+)$'
topic_rewrite_replacement '\1/rewritten'
<security>
use_tls true
<tls>
ca_file /path/to/aws/rootCA.crt
cert_file /path/to/aws/client.crt
key_file /path/to/aws/client.key
</tls>
</security>
<monitor>
send_time true
</monitor>
<buffer>
flush_interval 10s
</buffer>
</match>
</label>
---
During the test, input messages were sent by mosquitt_pub and output messages were confirmed by mosquitto_sub.
subscribe (confirm output)
mosquitto_sub --cafile aws/rootCA.crt --cert aws/client.crt --key aws/client.key -h "XXXXXXXXX.iot.ap-northeast-1.amazonaws.com" -p 8883 -q 1 -d -t test/rewritten -i clientid1
subscribe/publish (relay message)
fluentd -v -c fluent-sample.conf
publish (send a message)
mosquitto_pub --cafile aws/rootCA.crt --cert aws/client.crt --key aws/client.key -h "XXXXXXXXX.iot.ap-northeast-1.amazonaws.com" -p 8883 -q 1 -d -t test -i clientid4 -m "hogehoge"
During the errors you indicated, how do mosquitto_pub and mosquitto_sub behave? If the same kind of error produced by them, I guess that the root cause resides in GSM 800c driver. Please check the details with that community. If you think you need a patch to fluent-plugin-mqtt-io, your contribution is welcome :)
Best
