wal2json icon indicating copy to clipboard operation
wal2json copied to clipboard

Performance improvement after output refactoring

Open dpirotte opened this issue 5 years ago • 2 comments

In an attempt to bring our fork closer to master, we discovered that d4c0e814696695bbf853c48b38b7479e0f83f6c7 introduced a meaningful performance change. d4c0e814696695bbf853c48b38b7479e0f83f6c7 replaced several appendStringInfoString uses with appendStringInfo, which dramatically cleaned up the output code at the cost of a performance hit.

This proposal reduces decoding time by ~20% as measured by this benchmarking script. It swaps appendStringInfo for a new appendStringInfoStrings helper, which loops through a variable argument list and calls the more performant appendStringInfoString under the hood, skipping if the argument starts with a null byte, i.e. whitespace args when pretty-print isn't set. (I think this is safe in these use cases, and it is responsible for a full quarter of the performance return.)

I'm very open to suggestions or other approaches that keep the (very nice) output refactoring and reclaim most of the performance.

dpirotte avatar Nov 07 '18 06:11 dpirotte

I ran some additional benchmarks overnight with the same script to verify the performance delta, and it remains consistent.

pgbench txns rows output before after diff
100,000 1,200,057 15.219s 12.058s -20.8%
200,000 1,900,073 23.849s 18.227s -23.6%
300,000 2,600,083 31.222s 24.182s -22.6%
400,000 3,300,101 39.237s 31.146s -20.7%
500,000 4,000,111 47.055s 36.420s -22.7%

dpirotte avatar Nov 07 '18 15:11 dpirotte

@eulerto: this looks interesting. Thoughts?

jmealo avatar Apr 06 '19 20:04 jmealo