I installed the kube-prometheus-0.12 branch in version k8s1.26. I have prometheus in persistent storage, using nfs, but the startup error is reported,Need some help. Thank you

[houzhx759]

I installed the kube-prometheus-0.12 branch in version k8s1.26. I have prometheus in persistent storage, using nfs, but the startup error is reported,

Here is the error log:

root@test-01-0001 setup]# kubectl logs -f prometheus-k8s-0 -n monitoring

ts=2023-10-13T07:11:36.208Z caller=main.go:585 level=info msg="Starting Prometheus Server" mode=server Version = "(version = 2.46.0, branch = HEAD, revision = cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)"

T07:11:36.208Z caller=main.go:590 level=info build_context="(go=go1.20.6, platform=linux/amd64, user=root@42454fc0f41e, date=20230725-12:31:24, tags=netgo,builtinassets,stringlabels)"

ts=2023-10-13T07:11:36.208Z caller=main.go:591 level=info host_details="(Linux 5.8.7-1.el7.elrepo.x86_64 #1 SMP Fri Sep 4 13:11:18 EDT 2020 x86_64 prometheus-k8s-0 (none))"

ts=2023-10-13T07:11:36.208Z caller=main.go:592 level=info fd_limits="(soft=1048576, hard=1048576)"

ts=2023-10-13T07:11:36.208Z caller=main.go:593 level=info vm_limits="(soft=unlimited, hard=unlimited)"

ts=2023-10-13T07:11:36.224Z caller=web.go:563 level=info component=web msg="Start listening for connections" Address = 0.0.0.0:9090

ts=2023-10-13T07:11:36.225Z caller=main.go:1026 level=info msg="Starting TSDB..."

ts=2023-10-13T07:11:36.226Z caller=tls_config.go:274 level=info component=web msg="Listening on" address=[::]:9090

ts=2023-10-13T07:11:36.226Z caller=tls_config.go:313 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090

ts=2023-10-13T07:11:36.227Z caller= dir_lock. go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock

ts=2023-10-13T07:11:42.277Z caller=main.go:885 level=info msg="Stopping scrape discovery manager..."

ts=2023-10-13T07:11:42.277Z caller=main.go:899 level=info msg="Stopping notify discovery manager..."

ts=2023-10-13T07:11:42.277Z caller=manager.go:1023 level=info component="rule manager" msg="Stopping rule manager..."

ts=2023-10-13T07:11:42.277Z caller=manager.go:1033 level=info component="rule manager" msg="Rule manager stopped"

ts=2023-10-13T07:11:42.277Z caller=manager.go:1009 level=info component="rule manager" msg="Starting rule manager..."

ts=2023-10-13T07:11:42.277Z caller=main.go:936 level=info msg="Stopping scrape manager..."

ts=2023-10-13T07:11:42.277Z caller=main.go:895 level=info msg="Notify discovery manager stopped"

ts=2023-10-13T07:11:42.277Z caller=main.go:881 level=info msg="Scrape discovery manager stopped"

ts=2023-10-13T07:11:42.277Z caller=notifier.go:603 level=info component=notifier msg="Stopping notification manager..."

ts=2023-10-13T07:11:42.277Z caller=main.go:928 level=info msg="Scrape manager stopped"

ts=2023-10-13T07:11:42.277Z caller=main.go:1157 level=info msg="Notifier manager stopped"

ts=2023-10-13T07:11:42.277Z caller=main.go:1166 level=error err="opening storage failed: lock DB directory: no locks available"

Also, I directly modify statefulsets.apps to add parameters, this is not effective, where should I modify to add parameters?

Need some help. Thank you

[philipgough]

From https://prometheus.io/docs/prometheus/latest/storage/

CAUTION: Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may happen. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.

[houzhx759]

From https://prometheus.io/docs/prometheus/latest/storage/从 https://prometheus.io/docs/prometheus/latest/storage/

CAUTION: Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may happen. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.注意：Prometheus 的本地存储不支持不符合 POSIX 标准的文件系统，因为可能会发生不可恢复的损坏。不支持 NFS 文件系统（包括 AWS 的 EFS）。NFS 可能符合 POSIX，但大多数实现都不是。为了提高可靠性，强烈建议使用本地文件系统。

Thank you for answering my question, but I have no problem using NFS as the storage of pvc back end of prometheus in my old version, but I can't use it in the new version. I'm confused, and it doesn't take effect if I modify statefulsets directly. I want to add a few parameters, what should I do? Please help me. thank you

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[charles260]

从 https://prometheus.io/docs/prometheus/latest/storage/

注意：Prometheus 的本地存储不支持不符合 POSIX 的文件系统，因为可能会发生不可恢复的损坏。不支持 NFS 文件系统（包括 AWS 的 EFS）。NFS 可能符合 POSIX，但大多数实现都不符合。强烈建议使用本地文件系统以确保可靠性。

I am the same. In the previous version 0.11, it was also possible to use PVC to mount NFS, but in versions 0.12 and 0.13, once actively mounted, Pod cannot be started.