Support for Kafka Transactions

[thicks]

Feature Request

I'd like to see support for Kafka transactions added to the nestjs implementation.

Describe the solution you'd like

I'm using Kafka in production and am loving the elegance of the NestJS implementation. Now, I have use case that requires a more complicated use of Kafka - all or nothing processing of events. Kafka supports use cases like this through the use of transactions.

Through the producer config parameters, I can configure the client to support the use of transactions, but I have no means of accessing the actual producer to manage the transaction.

I'd like the ability to create/begin, commit and abort Kafka transactions.

[kamilmysliwiec]

For this, we would have to allow accessing customer and producer instances directly. Something to consider in the future.

[thicks]

yeah, that's what I see as well. I was able to extend and the factory and client to expose the transaction on the producer. I think I've been able to get a workable solution for the near term.

[iammathew]

As I really love Nest.js and work with Kafka i can speak out of experience, this is commonly needed. Especially when we consider the consume-transform-produce loop that needs this kind of feature. (Coming from a event sourcing/event driven architecture perspective this feature can also be hugely beneficial)

[HajoAhoMantila]

@thicks Do I understand your scenario correctly: this is about publishing multiple messages using ClientKafka.emit(), and having them published in a Kafka transaction?

I stumbled across this feature request when looking at a different scenario: in a request-response handler (i.e. annotated with @MessagePattern), I would like to be able to emit additional messages together with the response message, so that they are published in a Kafka transaction. For this to work, all messages would have to be sent using the same KafkaJS producer. It seems this would go against the design of the microservices extension, which hides the transport mechanism from the handler.

[thicks]

My opinions have evolved away from this request. Although Kafka is moving towards better support for transactions, I believe it’s a misuse of an event driven architecture. Transactions are best left to a model that’s more in line with RDB’s. I addressed our requirement with a better separation of concerns for our microservices. Your situation may be entirely different and lead you to different opinions.

[Papooch]

We are implementing idempotency and are using the transaction feature by extending ClientKafka (same as mr. thicks above) and exposing the producer instance.

@kamilmysliwiec I am interested in contributing the change upstream, would you greenligt a PR that adds the transaction feature to ClientKafka directly, or do you think it diverges too much from the general microservices API and should be provided by a 3rd party module?

EDIT: For context, the Kafka docs and a Confluent blog post

[kamilmysliwiec]

I think it may diverge too much from the general API (since other transporters don't support transactions) but still would love to see a PoC PR that showcases how much of an effort/maintenance burden that would be for us. If you're interested in creating a draft PR, definitely go for it @Papooch, but as I've just mentioned - not sure if we'll make the decision to merge it in.

[Papooch]

@kamilmysliwiec Here's the thing. I think that forcing the microservice API to be too general actually hurts the case. If the API was the same for all transports then there would be no reason to use one over any other, when in reality, using a certain transport is the result of some investigation and a bussiness decision. In that case I expect my framework of choice to help me integrate with it easily instead of making it difficult to actually use all the features that the transport offers.

That's why I don't buy the "since other transporters don't support transactions" argument. Of course they don't, that's why I chose to use this one.

The maintenance cost is a valid point though, but as long as NestJS only provides a simple adapter to the underlying library, it shouldn't be too hard.

I'll see if I can find time to create a draft PR.

[kamilmysliwiec]

I think that forcing the microservice API to be too general actually hurts the case.

That's the design decision we've made and so far it's working quite well, we don't plan to change it. If you need complete control over Kafka, you can just use kafkajs directly, there's no reason to create & maintain wrappers and abstractions for all underlying transporters. Using kafkajs directly without @nestjs/microservices is perfectly fine (and the same applies to other transporters). If, for some reason, you want to utilize NestJS abstractions, you can always create a custom transporter.

On the bright side, this conversation made me understand that creating a draft PR might indeed be a waste of time. Instead, for such use-cases, we should use custom transporters/or just use the library directly.

If you're willing to OSS your custom solution and share it with others, that would be great!