how to not load the history metrics from disk after resttart prometheus pod

[bruse-peng]

trafficstars

What did you do?

Did you expect to see some different?

Environment

• Prometheus Operator version:

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

insert Prometheus Operator logs relevant to the issue here

Anything else we need to know?:

my prometheus opreator metrics store on the disk, one day i delete the servicemonitor, 30 mintiues later i can still find the target on prometheus, then i restart the prometheus pod, the pod will load all history metric from disk then the kubernet node out of memory agian and again,so i have two issue: 1、why the tartget still can be find on prometheus when i delete servicemonitor 2、how to resolve not load all history metric from disk when i restart prometheus pod

[simonpasquier]

why the tartget still can be find on prometheus when i delete servicemonitor

without more details it's hard to tell. First thing would be to check the prometheus operator logs.

how to resolve not load all history metric from disk when i restart prometheus pod

I assume that you've configured persistent storage for Prometheus. You can configure an emptyDir volume instead if you don't need/want to retain data across restarts.

[bruse-peng]

configured persistent storage for Prometheus

yes i want persistent storage for Prometheus,but i configure an emptyDir volume, how can make sure data not be lost after i resstart prometheus pod

[simonpasquier]

I'm not sure to follow exactly but if you want persistent storage then read https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/storage.md.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.