KV store blocks on calls to get all keys

[segfaultdoc]

Problem

Calls to get all keys blocks even when there is a non-zero number of keys in the store. Note that this call always blocks when there are 0 keys in the store; not sure if the latter is desired behavior or not, however the former is def not desired.

The issue seems to be correlated with items being expired from the store or perhaps just generally the number of items decreasing.

So I'm pretty sure this is an issue with the server because the behavior is only reproducible on versions > 2.8.2.

Here's some code to reproduce: https://github.com/segfaultdoc/nats_blocking

Setup

Spin up v2.9.0 and run the above script. You'll notice logs stop getting printed after some time. I was not able to reproduce on versions 2.7.3 >= && <= 2.8.2

Running the script

export RUST_LOG=info
cargo run -- --nats-url nats://127.0.0.1:4222 --bucket-config-path ../bucket.yaml

[segfaultdoc]

Another issue I'm seeing when not allowing any items to TTL is the call to keys randomly returns 3 while I expect there to only be 2 keys in the cache at all times

[Jarema]

@segfaultdoc I'm looking into this.

[buffalu]

also worth noting it blocks if there's no keys in the bucket. imo it should not block and detect that somehow. we've hacked around it by fetching num_messages and only calling keys if num_messages > 0, but this is still vulnerable to race conditions

[Jarema]

That part might be a client bug. I'll check it an fix it if its the case.

[Jarema]

Fix for what @buffalu mentioned is already created: https://github.com/nats-io/nats.rs/pull/641 on Rust client.

Will look further into the main problem.

[buffalu]

nice thanks!