restarting nats streaming server client connections dont recover

[tanzeelrana]

On a fresh install when things are working fine we can see 7 clients and 12 channels(subscriptions)

After we restart the nats-streaming-server service these counts drop and some channels and client connections are unable to recover.

I added a connectionListener on all streaming connections to debug further, looking at the debug logs it looks like we get a disconnect -> connect -> resubscribed events in order when i restart the nats streaming server.

Observing the connection events being fired on a nats streaming server restart it looks like all client connections and subscriptions have recovered successfully but that is not the case as we have seen from the nats streaming server metrics and also grafana metrics.

I am using nats java stan library https://github.com/nats-io/stan.java

Does anyone know why my clients and subscriptions never come back even though the connection events say that everything is connected and resubscribed ?