Multiple read timeout reached errors

[yasin-amadmia-mck]

Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached

Getting loads of this. Any idea what could be wrong ? The AWS ES Cluster Status is GREEN.

I have

resurrect_after -1
reload_on_failure true
reload_connections false

[dhawal55]

+1

[pitr]

ES 2+ is not supported

[dhawal55]

2016-12-05 23:27:44 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:27:45 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] " plugin_id="object:1e5927c"
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:312:in `perform_request'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/ping.rb:20:in `ping'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:146:in `client'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:335:in `rescue in send'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:333:in `send'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:318:in `write'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/buffer.rb:354:in `write_chunk'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/buffer.rb:333:in `pop'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/output.rb:338:in `try_flush'
  2016-12-05 23:27:44 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/output.rb:149:in `run'
2016-12-05 23:27:45 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"xxx.es.amazonaws.com", :port=>443, :scheme=>"https", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="XXXXXX">, :region=>"us-west-2"}}
2016-12-05 23:27:45 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:27:47 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"message\":null}" plugin_id="object:1e5927c"
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:312:in `perform_request'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:334:in `send'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:318:in `write'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/buffer.rb:354:in `write_chunk'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/buffer.rb:333:in `pop'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/output.rb:338:in `try_flush'
  2016-12-05 23:27:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/output.rb:149:in `run'
2016-12-05 23:27:47 +0000 [warn]: retry succeeded. plugin_id="object:1e5927c"
2016-12-05 23:27:55 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:27:56 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"message\":null}" plugin_id="object:1e5927c"
  2016-12-05 23:27:55 +0000 [warn]: suppressed same stacktrace
2016-12-05 23:27:56 +0000 [warn]: retry succeeded. plugin_id="object:1e5927c"
2016-12-05 23:28:15 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:28:16 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"message\":null}" plugin_id="object:1e5927c"
  2016-12-05 23:28:15 +0000 [warn]: suppressed same stacktrace
2016-12-05 23:28:16 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:28:18 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"message\":null}" plugin_id="object:1e5927c"
  2016-12-05 23:28:16 +0000 [warn]: suppressed same stacktrace
2016-12-05 23:28:18 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:28:22 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"message\":null}" plugin_id="object:1e5927c"
  2016-12-05 23:28:18 +0000 [warn]: suppressed same stacktrace
2016-12-05 23:28:22 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-12-05 23:28:30 +0000 error_class="Elasticsearch::Transport::Transport::Errors::InternalServerError" error="[500] {\"message\":null}" plugin_id="object:1e5927c"
  2016-12-05 23:28:22 +0000 [warn]: suppressed same stacktrace
2016-12-05 23:28:31 +0000 [warn]: retry succeeded. plugin_id="object:1e5927c"

I'm using fluent-plugin-aws-elasticsearch-service v0.1.6 and the following config:

<match **>
    type aws-elasticsearch-service
    log_level info
    include_tag_key true
    logstash_format true
    logstash_prefix p15
    buffer_type memory
    buffer_chunk_limit 8m
    buffer_queue_limit 8192
    flush_interval 10s
    retry_limit 10
    disable_retry_limit 
    retry_wait 1s
    max_retry_wait 60s
    num_threads 1
    reload_connections false
    resurrect_after 5
    <endpoint>
      url https://xxxx.us-west-2.es.amazonaws.com
      region xxx
    </endpoint>
  </match>

[jasonhr13]

+1