cortex-helm-chart icon indicating copy to clipboard operation
cortex-helm-chart copied to clipboard

Published 20 hours ago • verified publisher icon cortexproject

Disable livenessprobe by default

Open friedrichg opened this issue 2 years ago • 1 comments
trafficstars

I have run all cortex components for years without liveness probes. they are bad for store-gateways because it causes re-sharding if they are under pressure. Restarting store-gateways can take a long time to recover, same for others components. We should disable all of them by default and let users enable them as needed.

Similar to https://github.com/cortexproject/cortex-helm-chart/pull/263

friedrichg avatar Oct 10 '23 07:10 friedrichg

Also bad for compactors as confirmed today

friedrichg avatar Feb 27 '24 12:02 friedrichg

they also seem to be designed wrong?

the ready endpoint for store-gateway returns

Some services are not Running:
Running: 3
Starting: 1

so store-gateway is being killed because some other service isnt ready, but the other service doesnt have a probe?

aep avatar Jun 25 '24 15:06 aep

+1

We should just remove it.

@friedrichg

Should we just drop livenessProbe for store-gateway and compactor then?

nschad avatar Jun 26 '24 07:06 nschad

@nschad yes, please

friedrichg avatar Jun 26 '24 08:06 friedrichg

Sorry for the hold-up

PR is open #502

@friedrichg

nschad avatar Jul 01 '24 06:07 nschad