nats-server

nats-server copied to clipboard

Feature Request

Similar to how a stream can be configured to bind multiple subject patterns, this request is to allow for a consumer to be configured with multiple filter subject patterns. Currently only one pattern can be specified.

Use Case:

My specific use case is modeling an event sourcing type of workload where I am modeling subjects as entities/state machines/workflows and there are a sequence of events written per subject. There are cases where I need to consume events from more than one subject in order to process a new command.

Currently, I am managing this by consume a broader scope of the stream, e.g. events.> and then doing local filtering. Supporting multiple filter subjects would make the consumers more targeted and reduce the data transferred and latency since it would be filtered server-side.

Also to clarify, this is all scoped to subjects in a single stream, not across streams, since I know that is a different problem (but interesting in its own right!)

Proposed Change:

To not break compatibility with the singular FilterSubject, this would add a new field FilterSubjects to take a slice of filter patterns.

Who Benefits From The Change(s)?

Anyone need to create a consumer with disjoint subjects, specifically not supported by wildcard hierarchy, e.g. []string{"foo.bar", "baz.qux"} or selecting specific subjects where a wildcard match would yield too much data, []string{"foo.1", "foo.2"} (where foo.* may have thousands or millions).

Alternative Approaches

The alternatives are either to do local filtering with a broader scoped filter or to creating multiple consumers per specific filter and combine the events/messages locally.

I think this is really needed, specially for messaging solutions

Any update on this please? Thank you.

Yes, we're looking into this. Expect more info soon.

Random idea on this topic: why not using Common Expression Language (cf. Go implementation) in order to filter out all kind of message based on their attributes?

# Filter a list of subject or subjects matching a regexp
msg.subject in ["foo.bar", "baz.qux"] || msg.subject.matches("(foo|bars?)_foo")

# Filter all messages sent at 10 AM with a particular header
msg.header["color"] == "blue" && msg.timestamp.getHours() == 10

This could also be extended to do some payload-based filtering with support for some serialization (JSON, protobuf, ...):

msg.payload.JSON().field1 == "foo" || msg.payload.protobuf().field3 == "bar"

This might be better served through our callout architecture which is underway. We prefer to try to keep the server as simple as we can.

Just found this issue. My own app architecture would also be greatly simplified if I could comma separate my interest declaration.

Any update on the roadmap for this?

@autodidaddict PR introducing this change already has been merged into dev branch and will be part od 2.10.0 release.

Additionally, async-nats has that feature released under server_2_10 feature flag (disabled by default until actual nats server 2.10 is released).

#3500 - implementation PR

Sweet thanks for the update

The title start with JetStream but is this only applicable to JetStream? I believe Nats Core also support this case, right?

The title start with JetStream but is this only applicable to JetStream? I believe Nats Core also support this case, right?

It only apply to jetstream since consumer is a notion of jetstream.

@autodidaddict PR introducing this change already has been merged into dev branch and will be part od 2.10.0 release.

Additionally, async-nats has that feature released under server_2_10 feature flag (disabled by default until actual nats server 2.10 is released).

#3500 - implementation PR

Since this issue is still open, this was not part of 2.10 I guess?

Yes this has been released in 2.10. Announcement post, docs, and 2.10 guide coming up next week.

Ah ok great. Thank you.