EventStore
Persistent Subscription stopped at last checkpoint
Describe the bug Persistent subscription Checkpoint is 950052 and Known is 1032562 Tried reconnecting Persistent Subscription, and it will not process any events.
Digging around, I can see the source stream is being written to (i.e. Known is going up)
The Checkpoint stream has this:
"950052"
Look at the source stream, I can see scavenged data. My guess is this is related to the problem:
I will probably write a new Checkpoint record to try and skip past this, but assuming the Evenstore should be able to handle this situation.
EventStore details
- EventStore server version: 21.10.1.0
Happened again, but noticed if I hit Replay Parked Message, it seems to wake up and start processing again. Not sure if that helps. We going to look at writing a process which checks all the persistent subscriptions and replays the parked queue each night to prevent our data being late.
@StevenBlair123 are there actually parked messages when you hit the replay parked messaage or is it empty ?
@ylorph - there were parked messages on all occassions (it's happened a few more times)
Can we try to see what happens if there are not parked messages? I know this can be a bit tricky to deliberately run into but I am curious what the behaviour would be.
When (if) the problem happens again, I will check each persistent subscriptions parked queue and report back.
Right, it's happened again:
The last checkpoint written looks like this:
Parked queue does have message (last written to on 6th April 2022 14:12:30)
Included the stream the persistent subscription reads from (no truncated events)
I am almost certain when I hit Replay Parked, this will magically fix itself, however, I can keep this in this state for a short time (before customer notices) if you need any more information.
here are all the log files. Couple of projections which had faulted (but were successfully restarted)