feat: add rdkafka support

[mkaufmaner]

Is there an existing issue that is already proposing this?

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe it

KafkaJS is no longer being actively maintained see https://github.com/tulios/kafkajs/issues/1603

Describe the solution you'd like

To use a library with librdkafka bindings instead of the pure javascript implementation provided by kafkajs.

See https://github.com/nestjs/nest/issues/12703#issuecomment-2084578236

To use the @confluentinc/kafka-javascript because Confluent Inc. now maintains the underlying librdkafka library. The @confluentinc/kafka-javascript may not be "production ready" but the developers on the project are actively working on getting it production ready.

Teachability, documentation, adoption, migration strategy

This new microservice transport option needs to be documented.

TODO: Expand upon this.

What is the motivation / use case for changing the behavior?

The motivation around this feature is to migrate towards a javascript kafka library that will be maintained in perpetuity.

[diepdang0810]

Do you have any updates? I can't wait for it

[un4ckn0wl3z]

Any update?

[Epyes]

Hi, do you have any update on this merge request ? We are currently facing this issue in my entreprise.

[Azuky]

Hey, I've seen that @confluentinc/kafka-javascript is now in "limited availability" instead of "early access". So it seems that now is more stable to be used and maybe now we can move forward with this migration.

Do you need help with something? I'm not an expert but I can try to assist if it's within my capabilities