nats-server tries to recover data for deleted JetStream account

[goku321]

trafficstars

nat-server version: 2.8.4

Steps to reproduce:

1. Create an account with JetStream enabled
2. Create a stream and publish some data
3. Delete the account by publishing to $SYS.REQ.CLAIMS.DELETE
4. Restart the nats-server

Even after the account is deleted nats-server tries to recover the JetStream data and then times out as it is not able to fetch the account. This can lead to multiple failed account lookups if the number of deleted accounts (with JetStream enabled) is high.

I've tried deleting all the streams and consumers by publishing to $JS.API.STREAM.DELETE.* and $JS.API.CONSUMER.DELETE.*.* respectively but that doesn't delete the entire store directory for an account. And, as long as the directory is present, nats-server will try to recover the JetStream data.

Proposal:

1. There should be a way to clean up the entire JetStream data directory for an account.
2. (OR) Account JWT deletion should also take care of deleting the JetStream data.
3. (OR) nats-server shouldn't try to recover JetStream data for a deleted account.

[derekcollison]

For now we suggest you reinstate the account, delete the streams, and the disable JetStream from the account, then re-remove.

We will improve this experience for sure but that should get you unstuck.

[matthiashanel]

@goku321 we will be adding your proposal 1, triggered by a system account message

[goku321]

Thanks @matthiashanel that would be great! For now, I'll try to include Derek's suggestion in our workflow

[goku321]

For now we suggest you reinstate the account, delete the streams, and the disable JetStream from the account, then re-remove.

We will improve this experience for sure but that should get you unstuck.

I tried this but no luck. I can still see the account fetch message in the logs. The only solution that works for me is to delete the account directory under the JetStream store directory. Maybe, I'm missing something?

[derekcollison]

After you re-instated the account you could see the fetch account being successful?

[goku321]

My bad. I actually deleted the account JWT along with deleting the streams and disabling JetStream. I thought I had to remove the account as well. I'll list the steps for clarity:

1. Delete the streams
2. Disable JetStream
3. Restart nats-server

Account fetch is successful and the server doesn't try to recover JetStream data.

Also, If I delete the account JWT after steps 1 and 2, nats-server still tries to fetch the deleted account which fails.

[derekcollison]

Thanks for the feedback.

[caleblloyd]

Fix is to issue the $JS.API.ACCOUNT.PURGE.<AID> endpoint to delete account data upon account deletion

Reference #3319