cloud-on-k8s
cloud-on-k8s copied to clipboard
elastic
Metric beats Error logs when following guide 1.3.0 (cannot connect to kibana)
Bug Report
What did you do? Follow the guide https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html on my minikube locally.
Installed:
- elastic search: quickstart
- kibana: quickstart
- file beat.
- Now tried setting up metricsbeat. (according to the example): https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration-examples.html
What did you expect to see?
Metrics in kibana coming in (but instead im only getting container logs from file beat)
What did you see instead? Under which circumstances?
The file beat works (running pod) but im not getting any metrics in Kibana.
In the logging i see:
2020-11-27T10:07:38.176Z ERROR instance/metrics.go:285 error getting group status: open /proc/450037/cgroup: no such file or directory
....
2020-11-27T10:07:38.176Z INFO instance/beat.go:452 metricbeat stopped.
2020-11-27T10:07:38.176Z ERROR instance/beat.go:956 Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).
Environment
-
ECK version:
1.3.0
-
Kubernetes information:
minikube version: v1.14.2 (on windows) commit: 2c82918e2347188e21c4e44c8056fc80408bce10
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
- Resource definition:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: quickstart
spec:
type: metricbeat
version: 7.10.0
elasticsearchRef:
name: quickstart
kibanaRef:
name: quickstart
config:
metricbeat:
autodiscover:
providers:
- hints:
default_config: {}
enabled: "true"
host: ${NODE_NAME}
type: kubernetes
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
process:
include_top_n:
by_cpu: 5
by_memory: 5
processes:
- .*
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event:
when:
regexp:
system:
filesystem:
mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
- module: kubernetes
period: 10s
host: ${NODE_NAME}
hosts:
- https://${NODE_NAME}:10250
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl:
verification_mode: none
metricsets:
- node
- system
- pod
- container
- volume
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: metricbeat
automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
containers:
- args:
- -e
- -c
- /etc/beat.yml
- -system.hostfs=/hostfs
name: metricbeat
volumeMounts:
- mountPath: /hostfs/sys/fs/cgroup
name: cgroup
- mountPath: /var/run/docker.sock
name: dockersock
- mountPath: /hostfs/proc
name: proc
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /sys/fs/cgroup
name: cgroup
- hostPath:
path: /var/run/docker.sock
name: dockersock
- hostPath:
path: /proc
name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
rules:
- apiGroups:
- ""
resources:
- nodes
- namespaces
- events
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- statefulsets
- deployments
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: default
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
- Logs:
2020-11-27T10:07:38.176Z ERROR instance/metrics.go:285 error getting group status: open /proc/450037/cgroup: no such file or directory
....
2020-11-27T10:07:38.176Z INFO instance/beat.go:452 metricbeat stopped.
2020-11-27T10:07:38.176Z ERROR instance/beat.go:956 Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://quickstart-kb-http.default.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [default-metricbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Basic realm=\\\"security\\\" ch... (truncated).
Im also hitting this with a similar setup (just changed the elasticsearchRef and bumped the version to 7.10.2
Hi, could you check if there's any error in the operator logs (kubectl -n elastic-system logs -f statefulset.apps/elastic-operator if installed with the quickstart) ?
It might be because the operator is not able to reconcile the association between Beat and Kibana.
+1 Also seeing this issue in a production cluster. It started happening after the cluster had been running for about three weeks, so it's unclear what cause this to fail
+1 here as well. New cluster running on Fedora Server 33. By chance, are any of the others hitting this issue also on Red Hat derivatives?
I had the same when anonymous user was enabled:
config:
xpack.security.authc:
anonymous:
roles: superuser
authz_exception: false
switching back to proper user auth solved the issue
Running Operator 1.5 on openshift 4.6.16 stack version 7.12 I am running into the same issue
{"level":"info","timestamp":"2021-04-27T16:26:17.061Z","logger":"index-management","caller":"idxmgmt/std.go:184","message":"Set output.elasticsearch.index to 'metricbeat-7.12.0' as ILM is enabled."} {"level":"warn","timestamp":"2021-04-27T16:26:17.061Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.061Z","caller":"eslegclient/connection.go:99","message":"elasticsearch url: https://elasticsearch-es-http.elastic-elk.svc:9200"} {"level":"info","timestamp":"2021-04-27T16:26:17.062Z","logger":"publisher","caller":"pipeline/module.go:113","message":"Beat name: metricbeat-beat-metricbeat-7cf49478cf-s24mx"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","logger":"monitoring","caller":"log/log.go:117","message":"Starting metrics logging every 30s"} {"level":"warn","timestamp":"2021-04-27T16:26:17.132Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","caller":"kibana/client.go:119","message":"Kibana url: https://kibana-kb-http.elastic-elk.svc:5601"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:152","message":"Total non-zero metrics","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":100000}},"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","stats":{"periods":3}},"cpuacct":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","total":{"ns":217798082}},"memory":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","mem":{"limit":{"bytes":1073741824},"usage":{"bytes":31682560}}}},"cpu":{"system":{"ticks":50,"time":{"ms":57}},"total":{"ticks":170,"time":{"ms":185},"value":0},"user":{"ticks":120,"time":{"ms":128}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"4cae33d5-afff-4f7c-b886-f0fc6a674c5a","uptime":{"ms":227}},"memstats":{"gc_next":16713344,"memory_alloc":14120648,"memory_sys":76104704,"memory_total":29426048,"rss":89141248... {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:153","message":"Uptime: 228.440641ms"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:130","message":"Stopping metrics logging."} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:465","message":"metricbeat stopped."} {"level":"error","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:971","message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {\"statusCode\":401,\"error\":\"Unauthorized\",\"message\":\"security_exception\"}."} Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"security_exception"}.
Noticed Metribeat successfully connects Kibana but after 2-4 restarts of the pod.
Running Operator 1.5 on openshift 4.6.16 stack version 7.12 I am running into the same issue
{"level":"info","timestamp":"2021-04-27T16:26:17.061Z","logger":"index-management","caller":"idxmgmt/std.go:184","message":"Set output.elasticsearch.index to 'metricbeat-7.12.0' as ILM is enabled."} {"level":"warn","timestamp":"2021-04-27T16:26:17.061Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.061Z","caller":"eslegclient/connection.go:99","message":"elasticsearch url: https://elasticsearch-es-http.elastic-elk.svc:9200"} {"level":"info","timestamp":"2021-04-27T16:26:17.062Z","logger":"publisher","caller":"pipeline/module.go:113","message":"Beat name: metricbeat-beat-metricbeat-7cf49478cf-s24mx"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","logger":"monitoring","caller":"log/log.go:117","message":"Starting metrics logging every 30s"} {"level":"warn","timestamp":"2021-04-27T16:26:17.132Z","logger":"cfgwarn","caller":"tlscommon/config.go:101","message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0"} {"level":"info","timestamp":"2021-04-27T16:26:17.132Z","caller":"kibana/client.go:119","message":"Kibana url: https://kibana-kb-http.elastic-elk.svc:5601"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:152","message":"Total non-zero metrics","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":100000}},"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","stats":{"periods":3}},"cpuacct":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","total":{"ns":217798082}},"memory":{"id":"crio-bc41e2251403c8c00b3cc983003aea6696a362531cb2ad6bb7dea0f771f1be8d.scope","mem":{"limit":{"bytes":1073741824},"usage":{"bytes":31682560}}}},"cpu":{"system":{"ticks":50,"time":{"ms":57}},"total":{"ticks":170,"time":{"ms":185},"value":0},"user":{"ticks":120,"time":{"ms":128}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"4cae33d5-afff-4f7c-b886-f0fc6a674c5a","uptime":{"ms":227}},"memstats":{"gc_next":16713344,"memory_alloc":14120648,"memory_sys":76104704,"memory_total":29426048,"rss":89141248... {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:153","message":"Uptime: 228.440641ms"} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","logger":"monitoring","caller":"log/log.go:130","message":"Stopping metrics logging."} {"level":"info","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:465","message":"metricbeat stopped."} {"level":"error","timestamp":"2021-04-27T16:26:17.207Z","caller":"instance/beat.go:971","message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {\"statusCode\":401,\"error\":\"Unauthorized\",\"message\":\"security_exception\"}."} Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-kb-http.elastic-elk.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"security_exception"}.
I had the same when anonymous user was enabled:
config: xpack.security.authc: anonymous: roles: superuser authz_exception: falseswitching back to proper user auth solved the issue
Hitting the same issue. Did you happen to find a resolution to this that doesn't involve removing anonymous authentication?
Im also experiencing the same issue in 8.2.2
{"log.level":"error","@timestamp":"2022-06-08T10:52:31.480Z","log.origin":{"file.name":"instance/beat.go","file.line":1038},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-prod-kb-http.elastic-system.svc:5601/api/status fails: [security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]: <nil>. Response: {\"statusCode\":401,\"error\":\"Unauthorized\",\"message\":\"[security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elas... (truncated).","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-prod-kb-http.elastic-system.svc:5601/api/status fails: [security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception: [security_exception] Reason: unable to authenticate user [elastic-system-metricbeat-beat-kb-user] for REST request [/_security/_authenticate]]: unable to authenticate user [elas... (truncated).
