#0 unexpected error error_class=RuntimeError error="No valid AWS credentials found.

[vishalmamidi]

my AWS OpenSearch setup only has "Fine-grained access control" so without AWS access key I should be able to access OpenSearch anywhere

Steps to replicate

    <match **>
      @type opensearch

      host "#{ENV['FLUENT_OPENSEARCH_HOST'] || 'localhost'}"
      port "#{ENV['FLUENT_OPENSEARCH_PORT']}"
      user "#{ENV['FLUENT_OPENSEARCH_USER']}"
      password "#{ENV['FLUENT_OPENSEARCH_PASSWORD']}"

      ssl_verify false
      logstash_format true
      reconnect_on_error true # defaults to false
      index_name fluentd-k8s
      type_name fluentd
    </match>

  <match **>
      @type opensearch
      user "xx"
      password xxxxxx
      ssl_verify false
      logstash_format true
      reconnect_on_error true
      index_name "fluentd-k8s"
      type_name fluentd
      <endpoint>
          url https://sxxcom:443
          region "ap-south-1"
      </endpoint>
  </match>

Using Fluentd and OpenSearch plugin versions

• OS version - ubuntu
• rancher-desktop Kubernetes
• Fluentd version v1.14.3
• OpenSearch plugin version 1.0.1

Fluentd boot logs

2022-02-06 19:57:55 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf" 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-avro' version '1.1.1' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-kafka' version '0.17.3' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.2' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.1' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-parser-avro' version '0.3.1' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0' 2022-02-06 19:57:55 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5' 2022-02-06 19:57:55 +0000 [info]: gem 'fluentd' version '1.14.3' 2022-02-06 19:57:55 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !! 2022-02-06 19:57:56 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead 2022-02-06 19:57:56 +0000 [info]: using configuration file: <ROOT>

<source>
    @type tail
    read_from_head true
    tag "kubernetes.*"
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    exclude_path ["/var/log/containers/fluent*"]
    <parse>
        @type "json"
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        unmatched_lines 
        time_type string
    </parse>
</source>
<filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
    kubernetes_url "https://10.43.0.1:443/api"
    ca_file ""
    skip_labels false
    skip_container_metadata false
    skip_master_url false
    skip_namespace_metadata false
</filter>
<match **>
    @type opensearch
    ssl_verify false
    logstash_format true
    reconnect_on_error true
    index_name "fluentd-k8s"
    type_name fluentd
    <endpoint>
      url https:xx.ap-south-1.es.amazonaws.com:443
      region "ap-south-1"
      user xx
      password xx
    </endpoint>
  </match>
</ROOT>

2022-02-06 19:57:56 +0000 [info]: starting fluentd-1.14.3 pid=7 ruby="2.6.9" 2022-02-06 19:57:56 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"] 2022-02-06 19:57:56 +0000 [info]: adding filter pattern="kubernetes." type="kubernetes_metadata" 2022-02-06 19:57:57 +0000 [warn]: #0 [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !! 2022-02-06 19:57:57 +0000 [info]: adding match pattern="" type="opensearch" 2022-02-06 19:58:09 +0000 [error]: #0 unexpected error error_class=RuntimeError error="No valid AWS credentials found." 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:246:in aws_credentials' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:601:in block in client' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/faraday-1.9.3/lib/faraday/connection.rb:91:in initialize' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/faraday-1.9.3/lib/faraday.rb:118:in new' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/faraday-1.9.3/lib/faraday.rb:118:in new' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/http/faraday.rb:72:in __build_connection' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:166:in block in __build_connections' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:158:in map' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:158:in __build_connections' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:69:in initialize' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:624:in new' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:624:in client' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:158:in __build_connections' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/opensearch-transport-1.0.0/lib/opensearch/transport/transport/base.rb:69:in initialize' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:624:in new' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:624:in client' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/opensearch_index_template.rb:62:in host_unreachable_exceptions' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/opensearch_index_template.rb:71:in rescue in retry_operate' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/opensearch_index_template.rb:69:in retry_operate' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:497:in handle_last_seen_os_major_version' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-opensearch-1.0.1/lib/fluent/plugin/out_opensearch.rb:380:in configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/plugin.rb:187:in configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:132:in add_match' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:74:in block in configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:64:in each' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/agent.rb:64:in configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/root_agent.rb:149:in configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/engine.rb:105:in configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/engine.rb:80:in run_configure' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/supervisor.rb:716:in block in run_worker' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/supervisor.rb:968:in main_process' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/supervisor.rb:708:in run_worker' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/lib/fluent/command/fluentd.rb:372:in <top (required)>' 2022-02-06 19:58:09 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in require' 2022-02-06 19:58:09 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in require' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.3/bin/fluentd:15:in <top (required)>' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd:23:in load' 2022-02-06 19:58:09 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd:23:in

' 2022-02-06 19:58:09 +0000 [info]: Worker 0 finished unexpectedly with status 1

[cosmo0920]

We’re using AWS-SDK and faraday middleware signv4 for accessing AWS OpenSearch Service. I’m not sure what is the correct fix for accessing to fine granted OpenSearch Service.

Patch is welcome.