parallel-consumer icon indicating copy to clipboard operation
parallel-consumer copied to clipboard

`checkAutoCommitIsDisabled` fails for `kafka-clients` v3.7.0

Open nscuro opened this issue 1 year ago • 1 comments

When using PC with kafka-clients v3.7.0, PC fails while trying to determine whether the provided Consumer has auto-commit enabled:

Caused by: java.lang.IllegalStateException: Cannot check auto commit is disabled for consumer type: org.apache.kafka.clients.consumer.KafkaConsumer
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.checkAutoCommitIsDisabled(AbstractParallelEoSStreamProcessor.java:493)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.validateConfiguration(AbstractParallelEoSStreamProcessor.java:330)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.<init>(AbstractParallelEoSStreamProcessor.java:283)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.<init>(AbstractParallelEoSStreamProcessor.java:266)
	at io.confluent.parallelconsumer.ParallelEoSStreamProcessor.<init>(ParallelEoSStreamProcessor.java:46)
	at io.confluent.parallelconsumer.ParallelStreamProcessor.createEosStreamProcessor(ParallelStreamProcessor.java:26)
...
Caused by: java.lang.NoSuchFieldException: coordinator
	at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
	at io.confluent.parallelconsumer.internal.AbstractParallelEoSStreamProcessor.checkAutoCommitIsDisabled(AbstractParallelEoSStreamProcessor.java:473)

Upon surface-level investigation, the failure is caused by https://issues.apache.org/jira/browse/KAFKA-15277. KafkaConsumer no longer holds a coordinator field.

Instead, KafkaConsumer now has a ConsumerDelegate<K, V> delegate field, which resolves to either LegacyKafkaConsumer, or AsyncKafkaConsumer.

LegacyKafkaConsumer still has a coordinator field, but AsyncKafkaConsumer does not.

nscuro avatar Feb 27 '24 10:02 nscuro

I will need to work on making the reflection checks to account for this, but i am thinking that it may be worth to add an override option to disable this check in edge cases like when custom delegate / wrapper for Consumer is used or when things break due to dependency changes until Parallel Consumer code catches up etc.

rkolesnev avatar Mar 04 '24 12:03 rkolesnev

Closing as fixed. Please re-open if still an issue after the #762.

Thanks @nscuro and @ondryaso!

rkolesnev avatar May 23 '24 08:05 rkolesnev