kube-prometheus Given the --web.enable-lifecycle prometheus, prometheus-config-reloader will still have failure logs

environment：k8s version -> 1.18,1.24,prometheus-operator version -> 0.62.0,prometheus-config-reloader version -> 0.62.0
prometheus version -> 2.41.0 logs: level=error ts=2024-02-27T06:54:56.669203009Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:01.665826086Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:06.664827358Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:11.668822581Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:16.66881128Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:21.668731898Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:26.665138413Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:31.669207027Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:36.669156825Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:41.669082053Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:46.667486132Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:51.669619723Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:55:56.668603295Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:01.665556184Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:06.664475166Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:11.669528246Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:16.669507555Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:21.668464686Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:26.669449591Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:31.6694677Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:36.669539578Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:41.66851047Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:46.669550155Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:51.668579207Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:56:56.668604976Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:01.668655775Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:06.668655133Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:11.665529792Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:16.668450138Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:21.669500204Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:26.668607056Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:31.665643856Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:36.668716044Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:41.668870103Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:46.668958682Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?" level=error ts=2024-02-27T06:57:51.66581294Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: received non-200 response: 401 Unauthorized; have you set --web.enable-lifecycle Prometheus flag?"
prometheus startup parameter '--web.config.file=/etc/prometheus/web_config.yaml' - '--config.file=/etc/prometheus/config_out/prometheus.env.yaml' - '--web.enable-lifecycle'

This results in an OOM in prometheus

Feb 27 '24 06:02 tjl-with-code

Have you solved this problem？

Oct 22 '24 09:10 wasd123qwe

hey I try to solve this.

I am using the helm chart without operator the one defined here https://github.com/prometheus-community/helm-charts

I define in the chart value.yaml this serverFiles

serverFiles:
  web_config.yml:
    basic_auth_users:
      admin: "bcrypt('of my password')"

and passing it here as extra args of configmapReload

https://github.com/prometheus-operator/prometheus-operator/blob/main/cmd/prometheus-config-reloader/main.go#L89-L92

configmapReload.prometheus:
    extraArgs:
      web-config-file: /etc/config/web_config.yml

But this does not work because "web-config-file" is the server side configuration. doing so enable server password for reloader.

other related issue https://github.com/prometheus-operator/prometheus-operator/issues/5836

Jan 06 '25 12:01 RouxAntoine