Query / Export Persisted Messages, Separate from a NATS Client

[sfrooster]

Persistent, ordered and highly-available messages enable a great many client use cases (support for recovery from client down time, cluster healing etc).

There are additional use cases where being able to (minimally) filter/query historic messages and export the results to pre-defined format in non-realtime (or near real-time if possible). For example:

• audit trail / compliance reporting
• production debugging
• alternate outcome testing
• shovel to a data pipeline / analytics / optimizing projections

First thoughts are you should be able to filter by topic, date range and sequence range.

[sfrooster]

This could be as easy as providing an alternative to deleting the oldest DAT file, and some CLI tooling to export all messages in a DAT to a stringified JSON array (file).

[aricart]

Would providing the format of the file be an useful feature? If the store files are large, it may be more useful to simply expose their format, and write tools against that.

[sfrooster]

The format(s) would be good, or a lib to write tools against, and then maybe a command line option for specifying an external tool to call at file switch, where it'd be passed some arguments like old file + path. I don't know. We'd have to talk about it, but the idea would be to override the default behavior of deleting the oldest file when the max number of messages is reached.

[sfrooster]

This is partially covered by #201