[BUG] org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses'

[devinbost]

Describe the bug We experienced an error that prevented us from consuming from Kafka on Pulsar into Apache Druid, and the only way around the problem was to switch back to using plain Kafka with Druid.

Here is the Druid log that contains the Kafka on Pulsar exception:

druid_log.txt

It would run about 2-5 minutes then throw that exception.

The consumption rate was also much slower than plain Kafka. About 200 messages/second vs 2000 messages/second. We tried a ton of consumer setting changes, but nothing seemed to help.

[sijie]

@jiazhai can you take a look at this issue?