goavro
goavro copied to clipboard
New option to reuse buffers when appending records
If you have similar record lists to append to a file, reusing some internal slices saves memory and seems to significantly improve the performances. On my computer:
Configuration | |||
---|---|---|---|
ReuseBuffers=false Deflate=false | 81_858_921 ns/op | 137_463_702 B/op | 10_341 allocs/op |
ReuseBuffers=true_Deflate=false | 19_568_794 ns/op | 21_023_761 B/op | 10_051 allocs/op |
ReuseBuffers=false_Deflate=true | 261_185_675 ns/op | 145_642_182 B/op | 10_520 allocs/op |
ReuseBuffers=true_Deflate=true | 122_100_900 ns/op | 9_794_948 B/op | 10_223 allocs/op |
The only strange thing is that reusing buffers takes less memory with deflate than without compression (but it is slower, which is consistent).
Do you think that it would interesting to add a such option?
Hi are you still there? Are you still interested in discussing this PR?
@xmcqueen Yes I'm still there. Do you have any remark/question?