Review producer API

[saraiva132]

At the moment we are only able to produce records one at a time and are also forced to flatten to "block" waiting for the broker ack if we do not want to take advantage of the asynchronous behaviour introduced by the exposed produce method.

It would be nice to change API to something like:

• produce(..) :F[Metadata]

• produceAsync(...) : F[F[Metadata]]

• produceChunk() : F[List[Metadata]]

• produceChunkAsync() : F[F[List[Metadata]]]

This is super pseudo code but I think the intent is here. WDYT ?

[x0ji0d]

What the bulked versions will return if only a subset of records succeeded?

[x0ji0d]

Also I like the "FF" versions more because otherwise I think most people will go with sync and don't look for how is it different from produceAsync. And flattening it is not a big deal if you need meta

[saraiva132]

I would say that it the expected behaviour is to either produce all records or fail all records.

I think both variants make sense because:

• They better express intent
• They give the user both options

Forcing people to go with double FF creates some confusion to some people, they do not quite understand why it is like it is and they will just flatten anyway. For that, might as well expose a sync method. Async is not that often useful anyway, only if you really want to leverage that ability to "block" waiting for the ack later on in your application.

[x0ji0d]

the expected behaviour is to either produce all records or fail all records.

For that you need to go with transactional producer, which I would not expect from plain produce

Async is not that often useful anyway, only if you really want to leverage that ability to "block" waiting for the ack later on in your application

My experience is quite opposite – I don't need to read meta in my app and the only "recovering" action is logging, which does not require blocking