NO_PROXY variable not honored

[waldner]

Running fluentd-kubernetes-daemonset:v1.14.6-debian-elasticsearch7-1.0 inside a k8s cluster behind a proxy. The container has the HTTP_PROXY/HTTPS_PROXY (and their lowercase version) variables set, which are honored; however, the proxy should not be used to connect to the k8s API, so I set NO_PROXY (and no_proxy) to kubernetes,10.43.0.1,kubernetes.default.svc, yet it looks like the API is not being accessed directly. Here are some errors from the log:

Successfully installed fluent-plugin-kubernetes-objects-1.1.12
1 gem installed
2022-07-20 13:13:02 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes-objects' version '1.1.12'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluentd' version '1.14.6'
2022-07-20 13:13:02 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
#<Thread:0x00007fe581a3da90 run> terminated with exception (report_on_exception is true):
/fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:87:in `rescue in start_pod_watch': start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://10.43.0.1:443/api: pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:myns:default\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:78:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:130:in `rescue in handle_exception': pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope (Kubeclient::HttpError)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:120:in `handle_exception'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 403 Forbidden (RestClient::Forbidden)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
	from /usr/local/lib/ruby/2.7.0/net/http.rb:933:in `start'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:352:in `block in get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:121:in `handle_exception'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'

On another cluster running without the proxy (everything else the same), no error is produced.