kafka-go

kafka-go copied to clipboard

Is there a way to set max.in.flight.requests.per.connection on the WriterConfig ?

https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#producerconfigs_max.in.flight.requests.per.connection

For us it is important since we need the order on a partition to be guaranteed. Kafka has some retry thing that can destroy the order if max.in.flight.requests.per.connection is larger then 1

Hello @josvegit, thanks for asking!

Currently, if you use the kafka.Reader, kafka.Writer, kafka.Client, or kafka.Transport, the kafka-go implementation will never perform more than one inflight request per connection. We made this decision because the multiplexing capabilities of Kafka are fairly limited. It is very easy to create head-of-line blocking situations where a blocking requests, or large responses prevents the processing of other inflight operations. With at most one request per connection, all requests can be processed concurrently without blocking each other.

Regarding max.in.flight.requests.per.connection, all retries are implemented client-side in Kafka (I'm not aware of any retry capabilities server-side, tho I could be about to learn something). In kafka-go, the kafka.Writer does not provide ordering guarantees, it is optimized for throughput and failure resilience instead. If you are looking to attain greater control over the dispatch of messages produced to partitions, I would advise you to look at using a kafka.Client instead (or the lower level kafka.Conn if you want to manage connections explicitly).

Ok good. I understand that the kafka.Writer does not guarantee any order. But kafka itself as I have understood guarantees that anything written to a partition within a topic has a guaranteed order. The order of how it was written. The retry is on the producer side you are right.

The documentation you referenced is from the Java client I believe, these settings do not exist in kafka-go.

Now kafka.Writer guarantee message order in partition?

In kafka-go, the kafka.Writer does not provide ordering guarantees, it is optimized for throughput and failure resilience instead

• Is it related to #686?
• Did the merge in #716 solve it?

Hello @josvegit

I wanted to follow up here and ask if the fixes suggested by @trevidawn had indeed address the original issue you reported.

Let me know!

@achille-roussel does kafka-writer provides any ordering guarantees now? If I have a loop that reads data from some source and pushes it to the same instance of the kafka writer - can I be sure that in relation to each partition those messages would arrive in order? (I do not care about ordering between partitions, and kafka doesn't guarantee it anyway).

I'm using writer with the following config:

&kafkago.Writer{
		Addr:            kafkago.TCP(brokers...),
		Topic:           topic,
		Balancer:        &kafkago.ReferenceHash{},
		BatchSize:       1,
		RequiredAcks:    kafkago.RequireAll,
		WriteBackoffMin: 0,
		Async:           false,
		AllowAutoTopicCreation: false,
}