orleans
orleans copied to clipboard
dotnet
Introducing NATS stream provider
This PR introduces a new stream provider for NATS.io.
The provider implementation follow the same approach as others which are based on queues. In this case, we use NATS JetStream stream as the backend "queue".
Each provider registration uses one NATS JetStream Stream. That stream is partitioned based on NatsOptions.PartitionCount using the Deterministic subject token partition making each pooling agent responsible for each partition.
Unlike all other providers, the state of the consumer is 100% managed by NATS Server. Consumers are named per partition, so if the partition owner changes, the next owner will continue to consume automatically. No need for external checkpoint storage. The provider acknowledge messages delivered upon delivery.
For now, it is implemented as a non-rewindable stream. We can later on enhance it to add rewind support, just need a bit more time and can be done on a separated PR.
Another area to investigate, is to use NATS KV for storage. In particular for PubSubStorage. The KV have interesting use cases like for example, external systems can be notified about state changes. That would allow people to self-contain the streaming without any other dependency, but this will go in another PR later.
Microsoft Reviewers: Open in CodeFlow
Will this be available for the 9.2.0 Release ? @ReubenBond I would love to have a streaming provider for Orleans which is not bound to the Azure environment.
Our team is also considering NATS as the stream provider.
I hope to see this on the next release
Hey @benjaminpetit! Did you had a chance to look at this?
We're already about to deploy a new system which would love to use this provider directly from the official packages rather then have its own fork like I'm doing with another project right now.
Please let me know any feedback you have so we can quickly fix it and move on. I'll rebase after the comments are addressed.
Thanks!
Just a heads up - There is a problem with the current version of NATS.Net package which does not allow the user to provide multiple JsonSerializerContext. Even tho the NatsOps allow it, it will stop seeking for source generated serializers on other JsonSerializerContexts registered on it. This will cause errors if the user use multiple contexts.
@mtmk is already aware of the situation and a new package should be released. It doesn't prevent the full review of this PR but please don't merge yet since I'll update to the new NATS.Net when it gets published.
@galvesribeiro I rebased the PR and added a commit to mark it as alpha for now. We can remove that after. Let me know when you're ready to proceed and if I need to update the NATS.Net package in the .NET nuget feeds
@galvesribeiro I rebased the PR and added a commit to mark it as alpha for now. We can remove that after. Let me know when you're ready to proceed and if I need to update the NATS.Net package in the .NET nuget feeds
Will check with the NATS folks and get back to you.
@ReubenBond I pushed the change with the new package (thanks @mtmk !) and updated the code to use it.
We need to update the package to NATS.Net 2.6.4.
Besides that, all good.
@galvesribeiro sorry, I merged a PR to convert to .slnx for solutions and now your PR has a conflict. Can you rebase? I think we can get this in.