Frequent leadership changes, using STAN and Jetstream

[juliev0]

HI, we are seeing very frequent leadership changes (every ~5 minutes) while running on GKE (but have heard it happens on EKS as well). I believe this happens whether we run with the NATS Streaming bus or Jetstream bus. And at the time the bus is basically idle.

If I change the MIN_ELECTION_TIMEOUT value from 500 msec to 1 sec they almost entirely go away (running with the Jetstream bus). The heartbeats go over NATS itself, right?

Any advice on what could be causing this? I don't believe we have any special configuration settings on NATS that would be causing any sort of rate limiting.