kube-prometheus icon indicating copy to clipboard operation
kube-prometheus copied to clipboard

The auto-discovery example can only be used in the default namespace, and other namespaces cannot get the target

Open AbelHugh opened this issue 1 year ago • 2 comments

kubernetes version: 1.24.4 kube-prometheus version: 0.12.0 doc site: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md

I use the sample yaml file to deploy, and the target can be obtained, but once the namespace is changed, the target cannot be obtained

example: example.zip

command: kubectl apply -n test -f example/ image image

log: prometheus-prometheus-0:

ts=2023-03-30T04:58:45.172Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.41.0, branch=HEAD, revision=c0d8a56c69014279464c0e15d8bfb0e153af0dab)"
ts=2023-03-30T04:58:45.172Z caller=main.go:561 level=info build_context="(go=go1.19.4, platform=linux/amd64, user=root@d20a03e77067, date=20221220-10:40:45)"
ts=2023-03-30T04:58:45.172Z caller=main.go:562 level=info host_details="(Linux 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 prometheus-prometheus-0 (none))"
ts=2023-03-30T04:58:45.172Z caller=main.go:563 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2023-03-30T04:58:45.172Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-03-30T04:58:45.176Z caller=web.go:559 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2023-03-30T04:58:45.179Z caller=main.go:993 level=info msg="Starting TSDB ..."
ts=2023-03-30T04:58:45.181Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090
ts=2023-03-30T04:58:45.182Z caller=tls_config.go:271 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2023-03-30T04:58:45.185Z caller=head.go:562 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-03-30T04:58:45.185Z caller=head.go:606 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.06µs
ts=2023-03-30T04:58:45.185Z caller=head.go:612 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-03-30T04:58:45.186Z caller=head.go:683 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2023-03-30T04:58:45.186Z caller=head.go:720 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=60.473µs wal_replay_duration=348.77µs wbl_replay_duration=260ns total_replay_duration=443.314µs
ts=2023-03-30T04:58:45.188Z caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC
ts=2023-03-30T04:58:45.188Z caller=main.go:1017 level=info msg="TSDB started"
ts=2023-03-30T04:58:45.188Z caller=main.go:1197 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
ts=2023-03-30T04:58:45.188Z caller=main.go:1234 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=701.457µs db_storage=1.88µs remote_storage=2.516µs web_handler=660ns query_engine=1.574µs scrape=347.226µs scrape_sd=8.356µs notify=1.553µs notify_sd=2.286µs rules=1.782µs tracing=10.443µs
ts=2023-03-30T04:58:45.189Z caller=main.go:978 level=info msg="Server is ready to receive web requests."
ts=2023-03-30T04:58:45.189Z caller=manager.go:953 level=info component="rule manager" msg="Starting rule manager..."
ts=2023-03-30T04:58:45.304Z caller=main.go:1197 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
ts=2023-03-30T04:58:45.305Z caller=kubernetes.go:326 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2023-03-30T04:58:45.306Z caller=main.go:1234 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=2.083612ms db_storage=2.757µs remote_storage=3.197µs web_handler=660ns query_engine=1.564µs scrape=62.536µs scrape_sd=722.977µs notify=1.9µs notify_sd=2.447µs rules=1.641µs tracing=6.01µs

prometheus-operator:

level=info ts=2023-03-30T04:33:13.189757699Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:33:13.945197669Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:33:53.249332081Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:33:54.15961701Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:33:54.730317282Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:33:55.818402681Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:33:55.818648532Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:33:56.670143784Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:35:12.866892193Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:35:13.761025384Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:35:16.065510828Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:35:16.954734787Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:35:28.163342175Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:35:29.04369118Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:35:29.949261824Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:35:31.042422567Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:36:52.04965227Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:36:52.941198914Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:36:58.768754919Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:36:59.64064407Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:38:10.868653308Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:38:11.711345397Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:38:12.115056466Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:38:13.046430368Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:38:19.212567755Z caller=operator.go:1162 component=prometheusoperator key=test/prometheus msg="sync prometheus"
level=info ts=2023-03-30T04:38:19.309360856Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:38:19.31012581Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:38:19.310172954Z caller=operator.go:1352 component=prometheusoperator key=test/prometheus statefulset=test/prometheus-prometheus shard=0 msg="not found"
level=info ts=2023-03-30T04:38:19.411969417Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:38:20.228355782Z caller=operator.go:638 component=alertmanageroperator key=monitoring/main msg="sync alertmanager"
level=info ts=2023-03-30T04:38:20.26647036Z caller=operator.go:1162 component=prometheusoperator key=test/prometheus msg="sync prometheus"
level=info ts=2023-03-30T04:38:20.266585559Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:38:20.436726158Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:38:20.436782745Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:38:21.458870828Z caller=operator.go:1162 component=prometheusoperator key=test/prometheus msg="sync prometheus"
level=info ts=2023-03-30T04:38:21.458997569Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:38:21.623593829Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:38:37.691713957Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:38:54.204061856Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:38:55.057162134Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:57:35.06013082Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:57:35.91794755Z caller=operator.go:638 component=alertmanageroperator key=monitoring/main msg="sync alertmanager"
level=info ts=2023-03-30T04:57:35.950015532Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:57:35.952389088Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:57:37.08091793Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:58:43.010515264Z caller=operator.go:1162 component=prometheusoperator key=test/prometheus msg="sync prometheus"
level=info ts=2023-03-30T04:58:43.20803598Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:58:43.211269609Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:58:43.232127371Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=error ts=2023-03-30T04:58:43.32480591Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="status \"test/prometheus\" failed: failed to update status subresource: Operation cannot be fulfilled on prometheuses.monitoring.coreos.com \"prometheus\": the object has been modified; please apply your changes to the latest version and try again"
level=info ts=2023-03-30T04:58:43.330135433Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:58:44.158449889Z caller=operator.go:638 component=alertmanageroperator key=monitoring/main msg="sync alertmanager"
level=info ts=2023-03-30T04:58:44.251903536Z caller=operator.go:1162 component=prometheusoperator key=test/prometheus msg="sync prometheus"
level=info ts=2023-03-30T04:58:44.252185061Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:58:44.428973352Z caller=operator.go:1162 component=prometheusoperator key=monitoring/k8s msg="sync prometheus"
level=info ts=2023-03-30T04:58:44.429085128Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:58:45.308688422Z caller=operator.go:1162 component=prometheusoperator key=test/prometheus msg="sync prometheus"
level=info ts=2023-03-30T04:58:45.308820633Z caller=operator.go:1330 component=prometheusoperator key=monitoring/k8s msg="update prometheus status"
level=info ts=2023-03-30T04:58:45.436044335Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"
level=info ts=2023-03-30T04:58:59.07385749Z caller=operator.go:1330 component=prometheusoperator key=test/prometheus msg="update prometheus status"

AbelHugh avatar Mar 30 '23 04:03 AbelHugh

Please refer to this document https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml

system51 avatar Apr 08 '23 04:04 system51

Please refer to this document https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleBindingSpecificNamespaces.yaml

it works, thank you.

AbelHugh avatar Apr 11 '23 01:04 AbelHugh