goavro icon indicating copy to clipboard operation
goavro copied to clipboard

New option to reuse buffers when appending records

Open Yves57 opened this issue 4 years ago • 2 comments

If you have similar record lists to append to a file, reusing some internal slices saves memory and seems to significantly improve the performances. On my computer:

Configuration
ReuseBuffers=false Deflate=false 81_858_921 ns/op 137_463_702 B/op 10_341 allocs/op
ReuseBuffers=true_Deflate=false 19_568_794 ns/op 21_023_761 B/op 10_051 allocs/op
ReuseBuffers=false_Deflate=true 261_185_675 ns/op 145_642_182 B/op 10_520 allocs/op
ReuseBuffers=true_Deflate=true 122_100_900 ns/op 9_794_948 B/op 10_223 allocs/op

The only strange thing is that reusing buffers takes less memory with deflate than without compression (but it is slower, which is consistent).

Do you think that it would interesting to add a such option?

Yves57 avatar Jan 22 '21 22:01 Yves57

Hi are you still there? Are you still interested in discussing this PR?

xmcqueen avatar Aug 19 '22 15:08 xmcqueen

@xmcqueen Yes I'm still there. Do you have any remark/question?

Yves57 avatar Sep 12 '22 20:09 Yves57