openwhisk-package-kafka
openwhisk-package-kafka copied to clipboard
Allow Kafka consumer configuration
Allow the feed subscriber to pass common consumer configuration options. https://github.com/openwhisk/openwhisk-package-kafka/blob/master/provider/consumer.py#L223-L236
Modifying the default consumer options enables lots of different usecases for this service. For example, I might want to turn off auto-commit or enable larger batch sizes for message polling.
This is a nice idea! One big problem it would solve is how to handle authenticated Kafka (not Message Hub) without overwhelming the trigger creator with a huge number of documented configuration options in the feed action.
We'll have to be careful though as something like auto-commit
is essential to the function of the feed service. Maybe the algorithm is to start with the config provided by the user and then overwrite that (if needed) with config values that are essential to the service.
As a short-term win, properties around message batch size would be great to start with.
fetch.min.bytes
fetch.max.bytes
max.poll.records