msbuild icon indicating copy to clipboard operation
msbuild copied to clipboard

Add Buffered BinaryReader to speed up Binlog reading

Open yuehuang010 opened this issue 1 year ago • 1 comments

WIP. Add Buffered BinaryReader to speed up Binlog reading. By directly buffering the stream, it can inline ReadByte() function into Read7BitEncodedInt(). Add BulkRead7BitEncodedInt() to read multiple numbers in one call.

yuehuang010 avatar Jul 31 '24 19:07 yuehuang010

Perf comparison would be nice then

I ported these changes to a private BinLogViewer to open a 500mb binlog.

Before: Total opening time: 1:11.670, Loading: 51.482 s, Analyzing: 8.743 s, Indexing: 7.944 s, Allocated: 120,133,521,336 bytes, Nodes: 143,137,731, Strings: 3,903,327

After: Total opening time: 52.185 s, Loading: 31.396 s, Analyzing: 9.339 s, Indexing: 7.796 s, Allocated: 63,378,355,056 bytes, Nodes: 143,137,731, Strings: 3,903,327

Note: This removes the bottleneck and exposes other places for further optimize. The gains are much greater in unit testing than End to End.

yuehuang010 avatar Aug 23 '24 03:08 yuehuang010