kafka-protocol-rs icon indicating copy to clipboard operation
kafka-protocol-rs copied to clipboard

RecordBatchEncoder::compute_size would be really nice

Open thedodd opened this issue 1 year ago • 2 comments

Thoughts on adding such a method? Similar interface to ::encode, but would calculate the encoding size based on the given params.

thedodd avatar Mar 05 '24 03:03 thedodd

Totally! These kinds of additional utilities are great. In general, it seems like you're using things a bit differently then others, and so some integration tests / examples of your patterns might be helpful here for others if you end up adding this method.

tychedelia avatar Mar 05 '24 22:03 tychedelia

One of the primary use cases would be for ensuring buffers have enough space to be able to encode a full record batch.

However, this might be tricky as compression will obviously influence this. Perhaps the best we could do is provide the size of the batch without compression, and make it very clear in the docs that if compression is used, the size will almost always be smaller.

thedodd avatar Mar 07 '24 19:03 thedodd