Faraday Error when using a new elastic search cluster

[mharun-tc]

When changing my elastic search cluster, I get the following error:

2022-06-01 14:23:41 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
2022-06-01 14:23:41 +0000 [info]: adding match pattern="**" type="copy"
2022-06-01 14:23:42 +0000 [error]: #0 unexpected error error_class=NameError error="uninitialized constant Faraday::Error::ConnectionFailed\nDid you mean?  Faraday::ConnectionFailed"
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.1.0/lib/elasticsearch/transport/transport/http/faraday.rb:63:in `host_unreachable_exceptions'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_index_template.rb:47:in `host_unreachable_exceptions'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_index_template.rb:57:in `rescue in retry_operate'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/elasticsearch_index_template.rb:55:in `retry_operate'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/out_elasticsearch.rb:484:in `handle_last_seen_es_major_version'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.2.2/lib/fluent/plugin/out_elasticsearch.rb:336:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/multi_output.rb:99:in `each'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/multi_output.rb:99:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin/out_copy.rb:39:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:132:in `add_match'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:719:in `block in run_worker'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'
  2022-06-01 14:23:42 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
2022-06-01 14:23:42 +0000 [error]: Worker 0 finished unexpectedly with status 1

With my old cluster I am not facing these issues anymore. The old cluster uses Elastic Search 7.16.1 while the new one uses 7.17.2.

The current docker file we are using for the custom fluentd image:

FROM fluent/fluentd:v1.14-debian-1

Use root account to use apt

USER root

below RUN includes plugin as examples elasticsearch is not required

you may customize including plugins as you wish

RUN buildDeps="sudo make gcc g++ libc-dev"
&& apt-get update
&& apt-get install -y --no-install-recommends $buildDeps
&& apt-get install -y libcurl4 libcurl4-openssl-dev
&& sudo gem install faraday_middleware
&& sudo gem install faraday-typhoeus
&& sudo gem install faraday-excon
&& sudo gem install elasticsearch -v 7.1.0 \ && sudo gem install fluent-plugin-elasticsearch
&& sudo gem install fluent-plugin-datadog
&& sudo gem uninstall faraday -v 2.3.0 --force
&& sudo gem install faraday -v 1.4.1
&& sudo gem sources --clear-all
&& SUDO_FORCE_REMOVE=yes
apt-get purge -y --auto-remove
-o APT::AutoRemove::RecommendsImportant=false
$buildDeps
&& rm -rf /var/lib/apt/lists/*
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems//cache/.gem

COPY fluent.conf /fluentd/etc/ COPY entrypoint.sh /bin/

USER fluent