Container crashes and keep restarting (splunk/kube-objects)

[whigley]

What happened: Setup splunk-kubernetes-objects for AWS EKS container crashes, see the log:

{"log":"2022-09-29 19:31:16 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"\n","stream":"stdout","time":"2022-09-29T19:31:16.151229669Z"} {"log":"2022-09-29 19:31:16 +0000 [info]: gem 'fluentd' version '1.15.1'\n","stream":"stdout","time":"2022-09-29T19:31:16.151269699Z"} {"log":"2022-09-29 19:31:16 +0000 [info]: gem 'fluent-plugin-jq' version '0.5.1'\n","stream":"stdout","time":"2022-09-29T19:31:16.151273479Z"} {"log":"2022-09-29 19:31:16 +0000 [info]: gem 'fluent-plugin-kubernetes-objects' version '1.2.0'\n","stream":"stdout","time":"2022-09-29T19:31:16.151278499Z"} {"log":"2022-09-29 19:31:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'\n","stream":"stdout","time":"2022-09-29T19:31:16.151280779Z"} {"log":"2022-09-29 19:31:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'\n","stream":"stdout","time":"2022-09-29T19:31:16.15131775Z"} {"log":"2022-09-29 19:31:16 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.0'\n","stream":"stdout","time":"2022-09-29T19:31:16.1513268Z"} {"log":"2022-09-29 19:31:16 +0000 [warn]: both of Plugin @id and path for \u003cstorage\u003e are not specified. Using on-memory store.\n","stream":"stdout","time":"2022-09-29T19:31:16.395608248Z"} {"log":"bundler: failed to load command: fluentd (/usr/bin/fluentd)\n","stream":"stderr","time":"2022-09-29T19:31:16.395726291Z"} {"log":"/usr/share/ruby/uri/rfc3986_parser.rb:67:in `split': bad URI(is not URI?): "https://fd2d:530c:e059::1:443/api" (URI::InvalidURIError)\n","stream":"stderr","time":"2022-09-29T19:31:16.395878724Z"}

What you expected to happen: To run and collect object logs to splunk

How to reproduce it (as minimally and precisely as possible): Setup values.yaml .. use default values

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.23
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release): Linux
• Splunk version: 9.0.1 (latest)
• Splunk Connect for Kubernetes helm chart version: version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}
• Others:

[hvaghani221]

Hi @whigley, thanks for reporting the issue. It looks like kubeclient is not able to parse the IPv6 host value. I have created an issue https://github.com/ManageIQ/kubeclient/issues/578

[hvaghani221]

Hi @whigley, as a workaround, you can temporarily add KUBERNETES_SERVICE_HOST, kube-objects plugin will generate URI using this env.

splunk-kubernetes-objects:
  enabled: true
  environmentVar:
    - name: KUBERNETES_SERVICE_HOST
      value: "[fd2d:530c:e059::1]"

[hvaghani221]

To fix the issue, we need to add logic to wrap the host value with [] if IPv6 is found here(https://github.com/splunk/fluent-plugin-kubernetes-objects/blob/521fb532027bbeb9423e29449092372828cc5a6d/lib/fluent/plugin/in_kubernetes_objects.rb#L142)

Ref: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/blob/db7fe0dd225d521c769902505ca09135aef7e71e/lib/fluent/plugin/filter_kubernetes_metadata.rb#L206-L208

[aryznar-splunk]

Close with workaround