Jetstream KV Cluster loosing data after nodes restart/

[grrrvahrrr]

Observed behavior

We start up a 5-node cluster with the following configuration and start to continuously put data into the KV:

port: 4222
http_port: 8222

cluster {
    name: js_kv
    listen: 0.0.0.0:6222
    connect_retries: -1
    pool_size: 9
    authorization {
        user: user
        password: password
        timeout: 0.5
    }
    routes = [
        nats-route://user:password@js_kv_node01:6222,
        nats-route://user:password@js_kv_node02:6222,
        nats-route://user:password@js_kv_node03:6222,
        nats-route://user:password@js_kv_node04:6222,
        nats-route://user:password@js_kv_node05:6222
    ]
    compression: {
      mode: s2_auto
      rtt_thresholds: [10ms, 50ms, 100ms]
    }
}

jetstream {
  store_dir: /data/jetstream
  max_file_store: 10737418240
}

To test sustainability we turn off two random nodes and wait for 5-10 minutes before turning them back on again. As a result after restart we see a difference in last sequence in the nodes that have been turned off. And in time it starts growing.

We also tried launching a durable consumer on the test cluster. As a result if the consumer leader switches from a healthy node to a node with sequence loss it stops delivering data due to unexpected sequence difference.

Following testing proves that in an unstable server environment kv can not guarantee the safety of data and getting data from it can give an unreliable result depending on which node is the cluster leader at the given moment.

Expected behavior

We expect that after turning nodes back on again, we would get the same last sequence on all 5 nodes and consumers would continue delivering data or at least alert that they have stopped doing so.

Server and client version

tested on both v2.10.14 and nightly-20240513 nats.go v1.34.1

Host environment

No response

Steps to reproduce

No response