Object deletion leads to unexpected result

[usatenko]

Defect

Make sure that these boxes are checked before submitting your issue -- thank you!

• [x] Included nats-server -DV output
• [ ] Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve)

Versions of nats-server and affected client libraries used:

nats-server: v2.9.15
[2453844] 2023/04/02 01:52:49.708242 [INF] Starting nats-server
[2453844] 2023/04/02 01:52:49.708771 [INF]   Version:  2.9.15
[2453844] 2023/04/02 01:52:49.708778 [INF]   Git:      [b91fa85]
[2453844] 2023/04/02 01:52:49.708780 [DBG]   Go build: go1.19.6
[2453844] 2023/04/02 01:52:49.708783 [INF]   Name:     NDCG6LAQLL76WOZJZLPHBVAECVITVRVPEYBQ2N2YCQ4HNBZKCYDTIB6H
[2453844] 2023/04/02 01:52:49.708786 [INF]   ID:       NDCG6LAQLL76WOZJZLPHBVAECVITVRVPEYBQ2N2YCQ4HNBZKCYDTIB6H
[2453844] 2023/04/02 01:52:49.708807 [DBG] Created system account: "$SYS"

OS/Container environment:

Linux ... 6.0.0-3-amd64 nats-io/nats-server#1 SMP PREEMPT_DYNAMIC Debian 6.0.7-1 (2022-11-05) x86_64 GNU/Linux

Steps or code to reproduce the issue:

nats object add a
nats object put a uhc-index.yml
nats object del a uhc-index.yml
nats stream view OBJ_a

Expected result:

I would like to see nothing, instead, I see a deleted message. The problem is that if I pass through thousands of objects add->del the number of such deleted messages is growing. I have not found how to delete them other than by calling something like this:

nats stream rmm OBJ_a "\\$O.a.M.>"

though I have not managed to make it to work. Also, the discard policy is set to "new", which leads to message accumulation, at some moment it will just stop receiving new messages (because deleted objects leave footprints as json with "deleted":true). I could be wrong but it is better to be set to "old".

Actual result:

{"name":"uhc-index.yml","options":{"max_chunk_size":131072},"bucket":"a","nuid":"C05VPWxFTFaHY4qZ533Bd9","size":0,"mtime":"0001-01-01T00:00:00Z","chunks":0,"deleted":true}

[derekcollison]

This is more about the Go client used in NATS cli vs server.

[usatenko]

I found a way to clean unnecessary messages with the command

nats stream purge OBJ_a --subject="\\$O.a.M.>"

So, the only thing is, there is an unobvious behavior, in case you put/del a lot of objects, it floods nats with deleted footprints, which at some point stops nats from being able to accept new objects.