Seahorse::Client::NetworkingError in logs

[SurenNihalani]

I keep seeing {"log":"2018-08-26 10:37:38 +0000 [warn]: #0 [out_cloudwatch_logs] failed to flush the buffer. retry_time=9 next_retry_seconds=2018-08-26 10:36:16 +0000 chunk="57453db16f8fbb08bbdcf68565f3c6e5" error_class=Seahorse::Client::NetworkingError error="execution expired"\n","stream":"stdout","time":"2018-08-26T10:37:38.189964632Z"} in fluentd's

I am using https://github.com/fluent/fluentd-kubernetes-daemonset v1.2

[SurenNihalani]

{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/net/http.rb:880:in `initialize'\n","stream":"stdout","time":"2018-08-27T01:05:16.531395431Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/net/http.rb:880:in `open'\n","stream":"stdout","time":"2018-08-27T01:05:16.531400457Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/net/http.rb:880:in `block in connect'\n","stream":"stdout","time":"2018-08-27T01:05:16.531404568Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/timeout.rb:101:in `timeout'\n","stream":"stdout","time":"2018-08-27T01:05:16.531407991Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/net/http.rb:878:in `connect'\n","stream":"stdout","time":"2018-08-27T01:05:16.531411366Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/net/http.rb:863:in `do_start'\n","stream":"stdout","time":"2018-08-27T01:05:16.531414755Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/net/http.rb:858:in `start'\n","stream":"stdout","time":"2018-08-27T01:05:16.531418063Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /usr/lib/ruby/2.3.0/delegate.rb:83:in `method_missing'\n","stream":"stdout","time":"2018-08-27T01:05:16.531421463Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/net_http/connection_pool.rb:288:in `start_session'\n","stream":"stdout","time":"2018-08-27T01:05:16.531424905Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/net_http/connection_pool.rb:95:in `session_for'\n","stream":"stdout","time":"2018-08-27T01:05:16.531471886Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/net_http/handler.rb:121:in `session'\n","stream":"stdout","time":"2018-08-27T01:05:16.531476561Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/net_http/handler.rb:73:in `transmit'\n","stream":"stdout","time":"2018-08-27T01:05:16.53147997Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/net_http/handler.rb:47:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531483566Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/plugins/content_length.rb:12:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531486817Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/json/error_handler.rb:8:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531490578Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/signature_v4.rb:66:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531493972Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531497511Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:92:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531501122Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:123:in `retry_request'\n","stream":"stdout","time":"2018-08-27T01:05:16.531504783Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:106:in `retry_if_possible'\n","stream":"stdout","time":"2018-08-27T01:05:16.531524809Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:94:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531528073Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:123:in `retry_request'\n","stream":"stdout","time":"2018-08-27T01:05:16.531531108Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:106:in `retry_if_possible'\n","stream":"stdout","time":"2018-08-27T01:05:16.531534673Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:94:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531537925Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:123:in `retry_request'\n","stream":"stdout","time":"2018-08-27T01:05:16.531541039Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:106:in `retry_if_possible'\n","stream":"stdout","time":"2018-08-27T01:05:16.531544099Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/retry_errors.rb:94:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.53154723Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/json/handler.rb:11:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531550253Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531553394Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/plugins/endpoint.rb:45:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531556311Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531559168Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531561972Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531564602Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531567565Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.53157045Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.531573564Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/plugins/response_target.rb:23:in `call'\n","stream":"stdout","time":"2018-08-27T01:05:16.53157667Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.20.2/lib/seahorse/client/request.rb:70:in `send_request'\n","stream":"stdout","time":"2018-08-27T01:05:16.531583092Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-cloudwatchlogs-1.3.0/lib/aws-sdk-cloudwatchlogs/client.rb:725:in `describe_log_groups'\n","stream":"stdout","time":"2018-08-27T01:05:16.531586187Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/out_cloudwatch_logs.rb:419:in `log_group_exists?'\n","stream":"stdout","time":"2018-08-27T01:05:16.531589215Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/out_cloudwatch_logs.rb:168:in `block in write'\n","stream":"stdout","time":"2018-08-27T01:05:16.531592355Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/out_cloudwatch_logs.rb:160:in `each'\n","stream":"stdout","time":"2018-08-27T01:05:16.531595523Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/out_cloudwatch_logs.rb:160:in `write'\n","stream":"stdout","time":"2018-08-27T01:05:16.531598681Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.1.3/lib/fluent/plugin/output.rb:1096:in `try_flush'\n","stream":"stdout","time":"2018-08-27T01:05:16.531601754Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.1.3/lib/fluent/plugin/output.rb:1329:in `flush_thread_run'\n","stream":"stdout","time":"2018-08-27T01:05:16.531604717Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.1.3/lib/fluent/plugin/output.rb:439:in `block (2 levels) in start'\n","stream":"stdout","time":"2018-08-27T01:05:16.531631484Z"}
{"log":"  2018-08-27 01:05:16 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.1.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'\n","stream":"stdout","time":"2018-08-27T01:05:16.531635815Z"}```

[meonthewire]

+1

[smurphnturf]

+1

[suryapssv]

@SurenNihalani @lmorais @smurphnturf Hi, is there any fix for the above error ?