openwhisk-package-kafka icon indicating copy to clipboard operation
openwhisk-package-kafka copied to clipboard

Allow Kafka consumer configuration

Open jthomas opened this issue 8 years ago • 2 comments

Allow the feed subscriber to pass common consumer configuration options. https://github.com/openwhisk/openwhisk-package-kafka/blob/master/provider/consumer.py#L223-L236

Modifying the default consumer options enables lots of different usecases for this service. For example, I might want to turn off auto-commit or enable larger batch sizes for message polling.

jthomas avatar Dec 02 '16 14:12 jthomas

This is a nice idea! One big problem it would solve is how to handle authenticated Kafka (not Message Hub) without overwhelming the trigger creator with a huge number of documented configuration options in the feed action.

We'll have to be careful though as something like auto-commit is essential to the function of the feed service. Maybe the algorithm is to start with the config provided by the user and then overwrite that (if needed) with config values that are essential to the service.

jberstler avatar Dec 02 '16 15:12 jberstler

As a short-term win, properties around message batch size would be great to start with.

fetch.min.bytes
fetch.max.bytes
max.poll.records

jthomas avatar Dec 02 '16 16:12 jthomas