Producer will persistently consume the memory via produce_no_ack

[belltoy]

When producing via produce_no_ack(fire & forget), and if the Kafka broker response is slower than producing, the producer will persistently consume the memory till OOM.

https://github.com/kafka4beam/brod/blob/master/src/brod_producer_buffer.erl#L118

It's better to add a threshold to limit that. Or drop messages when exceeding.

[zmstone]

Hi @belltoy

The API is designed to be async without back pressure.

maybe a configurable behavior:

• By default, keep the current behavior (backward compatible)
• Punishing (block) callers when buffer is full (this will need caller to use a different api)
• drop new no-ack requests if buffer is full (can’t drop old because otherwise we’d have to scan the buffer)

[zmstone]

A quick fix for you is maybe to make the caller alternatively use produce_no_ack and produce_sync e.g. call produce_sync once for every 100 produce_no_ack.

[belltoy]

Yes, I saw that's what this API was designed for. But need some method to handle this.

drop new no-ack requests if buffer is full (can’t drop old because otherwise we’d have to scan the buffer)

This idea maybe fit my scenario. But where to drop these requests?

The ack/no_ack term of this API is different from the acks term of Kafka, which is confusing. Or rather to say, like cast in Erlang. In the meanwhile, Kafka protocol acts a request/response model, except produce requests with RequiredAcks=0. There must be a response to a produce request, no matter what the required_acks is if RequiredAcks != 0.

People don't care about acks/response/others in this scenario, use acks=0, which means, produce/write if you can, or drop if not. But do not block/punish me.

EDIT: Produce request with RequiredAcks=0 is the only case where the server will not reply to.

[zmstone]

This idea maybe fit my scenario. But where to drop these requests?

This is an enhancement to be added.