aspnetcore

aspnetcore copied to clipboard

This issue captures issues related to the "eventing framework" work area in .NET 9. Issues and categorizations are subject to change as design and prototyping is underway.

This eventing framework will allow developers to write applications that support processing messages from various queue providers in their application.

Major categories of work include:

• Exposing APIs for registering event providers for various queue implementations, including a supported default set
• Exposing APIs for registering event handlers and implementing a routing implementation to dispatch events to the appropriate handler
• Expose APIs for serializing and deserializing messages resolved from providers
• Exposing APIs for exposing framework primitives like middlewares, filters, DI support, etc.
• Add support for managing the application host and pipeline
• Add support for relevant metrics/tracing in eventing framework

More details here:

https://github.com/dotnet/aspnetcore/issues/53219#issuecomment-1921960530

I'd like to chime in here a bit because I think a couple of points might be relevant to consider.

Right now there exist a couple of message handler frameworks. Some that come to mind are:

• https://github.com/jbogard/MediatR
• https://github.com/martinothamar/Mediator
• https://github.com/MassTransit/MassTransit
• https://github.com/JasperFx/wolverine

And many more exist.

Given that this is supposed to become a first party implementation of similar capabilities, is there going to be a call for usage scenarios so we can give feedback on which capabilities are used and how or is this going to be handled entirely internally?

I expect that this implementation will at least in part leverage source generation similar to how https://github.com/martinothamar/Mediator and https://github.com/JasperFx/wolverine do it. Albeit the difference between those two is that Wolverine does runtime source generation vs. Mediator using Roslyn source generators.

If it does use source generation I'd like to point out that special care should be taken looking at the integration with existing other source generators. Specifically because there still is no way to chain them together. It would be great if the capability to create MessageHandlers on the fly using Source generation would still allow them to be used from within whatever you're going to build. I'd love for the aspnetcore team to champion changes in Roslyn that lead to us being able to chain Source Generators :wink: . Given that the Razor generator already presents similar issues. Runtime code generation will likely not work that well together with AOT unless you do a two-pass system similar to how Wolverine handles it.

All in all I think having a first party implementation as something people can fall back on or that other libraries can expand on is a good thing. I'd just like it to be as usable as possible in the end :)

Isn´t this exactly what you already tried with WCF when it came out? A unified platform for uni and bidirectional communication.

What will you do differently this time around to prevent the mess that WCF was for, especially for one-way messaging?

I would request and highly recommend that in building this you introduce a common abstraction layer that can be used by third party tools, much like the ILogger abstraction introduced in previous versions of .NET.

Why not support existing OSS tools like Wolverine or MediatR or smth else instead of building smth from scratch in-house?

One question pops into my mind: How are schema registries covered?

Is "Expose APIs for serializing and deserializing messages resolved from providers" meant to support schema registries out of the box, or is is meant to provide an interface to integrate a schema registry on your own?

This is interesting. Some thoughts from experience with existing libraries in the space.

• It would be great to be able to specify the "shape" of the handlers / destination e.g. route to methods on this interface. A big barrier to adoption of existing libraries is that they force you to change all your existing handlers to their custom interface & all change all your pocos to inherit from some base class. It can mean changing a lot of code. A more lightweight approach would be wonderful.

• Please support (or don't somehow exclude) databases as event sources. Many applications would be just fine with a database backed queue (hello SKIP LOCKED in postgres). And using the database has other advantages including a natural outbox implementation, dead simple local development experience and more. It's a great place to start as long as one can switch to a proper queue down the road.

With outbox feature, please.

Why not support existing OSS tools like Wolverine or MediatR or smth else instead of building smth from scratch in-house?

One man library. The tools are designed to promote their creators. The design and functionality depend on the owner. When someone is dissatisfied, they create another very similar tool to fulfil their needs.

Create APIs and Layers to standardise these tools seems much better to me.

Why not support existing OSS tools like Wolverine or MediatR or smth else instead of building smth from scratch in-house?

One man library. The tools are designed to promote their creators. The design and functionality depend on the owner. When someone is dissatisfied, they create another very similar tool to fulfil their needs.

Create APIs and Layers to standardise these tools seems much better to me.

Imo, that's exactly against the spirit and nature of OSS. Tools are not designed to promote developers(while this might be a good side effect). Those tools are built to solve problems, and people collaborate together to solve them. Now, MS looks at the existing ecosystem selects popular areas, and develops tools that essentially replace existing OSS alternatives. So what happens next? Use of those tools will obviously drop, and eventually, they will disappear. Why maintain some OSS tool when you can just sit, wait, and hope MS will be kind enough to develop the thing that solves your problem? This is not OSS.

Just as a point of clarification, MediatR is NOT comparable to this feature set, nor would plug in to a common event bus interface.

MediatR is strictly in-process and in-memory. Not for durable async message queues.

And also evidently MediatR exists solely to promote myself.

I like this idea though. It should be easier to adopt event-driven architectures. It would be nice if it were like DI, logging, even EF Core where there is some default implementation but you can add your own 3rd party implementation.

The risk is that, similar to what happened with DI, the "common interface" effectively defines the implementation and stifles innovation. There was a great thread many moons ago where the author of Simple Injector was trying to explain why the DI interface was hostile to the fundamental premise of Simple Injector. IIRC the conclusion was ~"this is a solved problem that doesn't need innovation" which was off base considering Simple Injector was a reasonably popular project that was taking a different tack that wasn't achievable with the "common interface".

Not sure how that all finally resolved in the end.

My desire would be to always be able to opt out and also to be able to rebuild whatever is given from smaller bits of the framework.

One man library. The tools are designed to promote their creators. The design and functionality depend on the owner. When someone is dissatisfied, they create another very similar tool to fulfil their needs.

FWIW, there's a company behind Wolverine (JasperFx), with similar commercial support offerings behind NServiceBus, MassTransit, and Rebus to name a few others. That argument doesn't really fly. Even Wolverine as the newest kid on the block has technical roots going back a decade. NServiceBus and MassTransit are ~15 years old at least.

Worth noting if you find this exciting, that we already offer all the features described in Brighter (and more @voroninp we have your outbox and inbox already):

https://github.com/BrighterCommand/Brighter

So if you are looking to work with a mature messaging framework, battle-tested at scale, (that also supports .NET Framework) feel free to check us out and start using events today instead of waiting.

We have been doing this for 10+ years, so our offering comes with a wealth of experience.

But to the larger point. The .NET space is perhaps more richly served than any other ecosystem with messaging tools, both commercial, commercial support and solely OSS. Competition amongst those projects has created a strength in offerings that other languages don't have.

When MS enters the space that diversity and competition will be crushed, leading to less innovation and choice.

There is no argument that MS needed to enter this space, .NET developers were more richly served than any other ecosystem.

Instead this is just the usual MS playbook - crush successful OSS - own all the things.

When I spoke about a .NET Renaissance many years ago I was clear that MS had a duty of care as the 500lbs gorilla in the room not to do this.

But whilst previous flattening of .NET OSS could be seen as ignorance or lack of care, they know well enough by now, that this can only be seen as malice.

FWIW, I'm voting hard against any set of common abstractions for the entry point to messaging, and even more so to any kind of standardized interface signature for message handlers. That's a recipe for a mediocre developer experience that won't make .NET "the best platform for cloud native applications". This isn't a place for a "Conforming Container" kind of approach.

Please, No.

We are thankfully blessed with a lot of good libraries at various layers for event-ish type stuff:

• MagicOnion
• GRPC
• NATS
• Orleans (if you look at it right and/or with the right level of violence)
• Akka.Net (if you look at it right and/or with the right level of violence)
• MessagePipe (if it's still around, I loved it though)
• MediatR
• RX and RX.Net
  • Also cysharp's R3
• MassTransit
• All the rabbit-MQ stuff
• Lots of other great things I don't know about.

Again, all of the above are extremely useful in the context of eventing or event driven systems.

I don't see how one could make a sane abstraction however, since many of them are context dependent yet cannot be easily simplified. They mostly have enough different but unique semantics to where a common abstraction could easily be either of little utility or just dangerous.

My apologies for any firmness here, I have concerns that another 'conforming container' across all of these paradigms would risk turning the story of event driven systems in .NET into a cautionary tale about good intentions stifling innovation and flexibility, potentially at the benefit of specific vendors despite the unique opportunities each of the above mentioned cases would handle.

I hope the community does not decide to add a 'conforming container, but for events', especially if it happens to conform best to a brand new system without clear benefit compared to existing ones aside from 'being first party' or 'well you need it anyway because ASPNETCORE'.

If we wanted to fix eventing at a better level, I would suggest:

• Working to provide wrapping invokers for events that allow for easy use of Task/ValueTask with various semantics and minimizing allocation.
• Providing a first party abstraction to get as close to I want everything directly written in this async method to execute on a dedicated thread (or within a limited priority pool) as possible.
• Providing FLEXIBLE (i.e. easy but able to get unsafe-ish) Option and Either ValueTypes.
  • One can worry about CLR opt later, I just hate juggling namespaces in FP codebases 😅

Points One and Three, I think are fairly easy, it's just a matter of putting them where people can get to them.

That middle point, It actually solves a lot of other problems in the .NET space...

More Importantly, It helps the authors of all of the above things give better results to the community, as they have done for so long.

If you -do- choose to go down the route of a common abstraction, I would hope that it is a public process and you involve the above mentioned parties as well as others.

Just keeping an eye on this as a maintainer / co-creator of https://github.com/FoundatioFx/Foundatio. I just hope this doesn't follow the path of IDistributedCacheClient that I've seen used by very few. Please engage early and often with leaders who have been in this space a long time.

Providing a first party abstraction to get as close to I want everything directly written in this async method to execute on a dedicated thread (or within a limited priority pool) as possible.

That is definitely an interesting one. Brighter uses a reactor/proactive model (depending on whether you use async) and has a single threaded message pump to preserve ordering (which you scale out). This also means we don't run into problems with thread pool exhaustion, or blocking on semaphores trying to limit work in flight at high scale. We use a custom SerializationContext based on an article by Stephen Toub to have async callbacks use that thread. But is a little gnarly and a framework solution to that problem would have re-usable value.

Just keeping an eye on this as a maintainer / co-creator of https://github.com/FoundatioFx/Foundatio. I just hope this doesn't follow the path of IDistributedCacheClient that I've seen used by very few. Please engage early and often with leaders who have been in this space a long time.

I love foundatio ❤️❤️❤️

Maybe one could compare the amount of money & effort that would be needed to develop a de novo fit-for-use eventing platform against the money & effort needed to create a set of template projects?

For Event Driven Architectures, having the right set of templates may be 80% of the developer win.

If this does move forward, please also add:

• Exposing APIs for managing dead-letter queues.

With some solutions, it can be hard to even figure out what the DLQ is named...

Also, Kafka should be considered as it greatly impacts design.

Also, Kafka should be considered as it greatly impacts design.

This proposal is about message queues. Kafka is not a queue, I don't think it should be considered.

Good article about Kafka and message buses: https://particular.net/blog/lets-talk-about-kafka

Queues vs Streams: https://youtu.be/RVmDU1QJmAs?si=zNA2EJslnvX23-mg

This more just comes down to messaging.. and I'm leaving this here https://x.com/BdKozlovski/status/1735344274655592685?s=20

Wanna use Kafka as a simple Queue?

With the new KIP-932: Queues for Kafka, you soon may be able to!

First, apologies for the confusion and concern we caused with this issue.

The idea was to focus on small subset of event handling, effectively "WebJobs v2" (https://github.com/Azure/azure-webjobs-sdk), principally for the Azure Functions folks but generic enough for everyone (like WebJobs is today). The goal is to align the Functions programming model with ASP.NET Core Minimal APIs so the developer experience is consistent. We have no plans to implement the application-level patterns which the rich ecosystem of .NET messaging frameworks and libraries already implement, such as sagas, outbox, persistence, or DTC integration. Your feedback here is important to us, and we will do better to reduce ambiguity up-front in the future.

Here are some more details:

• We're building on top of CloudEvents (https://cloudevents.io/) as the core event-type primitive in this system to provide a good integration story with existing systems and to build on established standards for cloud native applications.

• We want to build a framework that follows the same paradigms and style as ASP.NET Core. This means building on top Microsoft.Extensions.*, and adding concepts like middleware, filters, and minimal APIs for events.

I want to be clear that this is not a work area that's been approached with malice towards or ignorance of the various libraries that exist in the ecosystem. Figuring out when we should integrate with/recommend/use/etc. an existing package when we're exploring a particular problem area is always complex, and we don't take that responsibility lightly. Our overarching goal here is always to be responsible towards our users and our community.

We're still prototyping and figuring out details, but this is what we sketched out:

var builder = Host.CreateApplicationBuilder();

builder.AddAzureQueueService("orders");

builder.Services
  .AddEvents()
  .AddAzureStorageQueueEventProvider("orders")
  .AddTimerEventProvider("cron", "*/5 * * * *");  

var app = builder.Build();

app.UseRouting();
app.UseExceptionHandler();

var orders = app.WithProvider("orders");
var cron = app.WithProvider("cron");

orders.MapEvent("order-received", (Order order, ILogger<Program> logger) =>
{
  // Handler logic here
});

cron.MapEvent((TimerInfo timer) =>
{
  // Handler logic here
});

app.Run();

Thanks for the feedback @davidfowl it is helpful

@davidfowl Would I be right in thinking then that you perceive this as closer in scope to Celery than a messaging framework? https://docs.celeryq.dev/en/stable/getting-started/introduction.html

Why are you calling it eventing, when it's clearly about messaging? Think about the confusion you'll introduce with the DDD crowd