Producer property message.max.bytes should be max.request.size?

[hackery]

While investigating issues around message size limits, we found that using message.max.bytes was necessary in the producer client to limit outgoing batch sizes, even though this is only documented as a broker property. The Java/Scala producer code does not use this property (except in module tests). The correct producer property for this function seems to be max.request.size.

• [x] librdkafka version (release number or git tag): v1.3.0
• [x] Apache Kafka version: Confluent 5.4.0
• [x] librdkafka client configuration: debug=all,message.max.bytes=500000
• [x] Operating system: CentOS 7.7.1908 (x64)>
• [ ] Provide logs (with debug=.. as necessary) from librdkafka N/A
• [ ] Provide broker log excerpts N/A
• [ ] Critical issue

[edenhill]

As a counterpoint I'd say that max.request.size is ambigiously named, the name suggests it applies to any protocol-level request, but it also applies to batch sizes and individual message sizes.

I think it makes more sense to have message.max.bytes also on the producer having the same meaning (more or less) as the broker setting with the same name, there is symmetry in that.

[hackery]

I think it's fair to say there's some ambiguity of naming in the Kafka documentation, and maybe this symmetry is something they should consider, but diverging from the documented behaviour as it stands seems a source of confusion. Maybe it would be worth at least noting in the lidrdkafka config docs?

[edenhill]

Part of the problem is that librdkafka predates the Java producer, but what we can do is add an alias max.request.size -> message.max.bytes.

[ijuma]

I agree that the Java producer config name is confusing. It also checks the size before compression.