Dapr Kafka Bulk messaging enhancement

[rahulpoddar-fyndna]

Dapr bulk messaging requires producer of the message to bulk it and send to side car in the application as of now. I could not find parameter setting such as linger.ms/batch.size/compression in yaml configuration. The pure java SDK based kafka client has a way to set the following properties

1. linger.ms
2. batch.size
3. compression

We are requesting to add support for above parameters so that the application code does not have to bother about bulking within a given time limit and create bulk messages in a for loop. Compression will further help in improving the performance. This needs to be taken care by underlying SDK as is the case with java SDK for apache kafka.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[Gallardot]

The sarama kafka client can only set linger.ms, batch.size, and compression for producer instances and not at the topic level.

In Dapr, a component instance corresponds to a producer. Therefore, linger.ms, batch.size, and compression can only be set at the component level.

Does this sound appropriate? If so, I might attempt to implement this feature.

@artursouza @rahulpoddar-fyndna WDYT?