Kafka Listener Batch Consumption - Need Kafka Keys

[arudge]

Expected Behavior

When using kafka batch processing, it should be possible to get the list of @KafkaKey associated with each message. I've tried two different ways but both do not seem to work.

1. @KafkaKey List<String> keys, List<Book> books
2. List<ConsumerRecord<String, Book>> bookRecords

Actual Behaviour

Here are two spock tests documenting the expected and actual behavior: https://github.com/arudge/micronaut-kafka/commit/946d712a46e1281f70ae06d17679b5078575e850

They provide more details on the conditions and exceptions seen. Here's a summary from the commit:

1. @KafkaKey List<String> keys, List<Book> books Provides a concatenated list of @KafaKey for each message.
2. List<ConsumerRecord<String, Book>> bookRecords Throws an exception trying to deserialize.

Steps To Reproduce

Run the spock tests added here: https://github.com/arudge/micronaut-kafka/commit/946d712a46e1281f70ae06d17679b5078575e850#diff-955cc0212d81c274ec6fcce11928af35344190843556416f41c32d758b5d0182R59

Environment Information

OSX

Example Application

https://github.com/arudge/micronaut-kafka/commit/946d712a46e1281f70ae06d17679b5078575e850#diff-955cc0212d81c274ec6fcce11928af35344190843556416f41c32d758b5d0182R59

Version

4.1.2-SNAPSHOT

[jonas-grgt]

I don't think receiving the key's like that is supported or would be a good design as the key and message body belong together. Also from a usability perspective you would always have to correlate the item in the key list with the correct body element.

Maybe you could directly receive or use a List<ConsumerRecord<KeyType, BodyType>> in your listener method ? But I'm not sure that is supported.

[arudge]

I don't think receiving the key's like that is supported or would be a good design as the key and message body belong together. Also from a usability perspective you would always have to correlate the item in the key list with the correct body element.

I totally agree and would prefer a List<ConsumerRecord....

Maybe you could directly receive or use a List<ConsumerRecord<KeyType, BodyType>> in your listener method ? But I'm not sure that is supported.

I tried that as well (see #2 above there).

[jonas-grgt]

I tried that as well (see #2 above there).

I don't see any listeners declared having a type List<ConsumerRecord<KeyType, BodyType>> as parameter ?

[arudge]

I don't see any listeners declared having a type List<ConsumerRecord<KeyType, BodyType>> as parameter ?

Here it is: https://github.com/arudge/micronaut-kafka/commit/946d712a46e1281f70ae06d17679b5078575e850#diff-955cc0212d81c274ec6fcce11928af35344190843556416f41c32d758b5d0182R339

and the corresponding test with the exception output: https://github.com/arudge/micronaut-kafka/commit/946d712a46e1281f70ae06d17679b5078575e850#diff-955cc0212d81c274ec6fcce11928af35344190843556416f41c32d758b5d0182R62

[graemerocher]

Duplicates #502

[guillermocalvo]

Duplicated by #502 Solved by #598

@arudge Please note that the supported way to receive keys in batch mode is via List<ConsumerRecord<String, Book>>, as documented in the Micronaut Kafka guide.