wal2json
wal2json copied to clipboard
Performance improvement after output refactoring
In an attempt to bring our fork closer to master, we discovered that d4c0e814696695bbf853c48b38b7479e0f83f6c7 introduced a meaningful performance change. d4c0e814696695bbf853c48b38b7479e0f83f6c7 replaced several appendStringInfoString uses with appendStringInfo, which dramatically cleaned up the output code at the cost of a performance hit.
This proposal reduces decoding time by ~20% as measured by this benchmarking script. It swaps appendStringInfo
for a new appendStringInfoStrings
helper, which loops through a variable argument list and calls the more performant appendStringInfoString
under the hood, skipping if the argument starts with a null byte, i.e. whitespace args when pretty-print
isn't set. (I think this is safe in these use cases, and it is responsible for a full quarter of the performance return.)
I'm very open to suggestions or other approaches that keep the (very nice) output refactoring and reclaim most of the performance.
I ran some additional benchmarks overnight with the same script to verify the performance delta, and it remains consistent.
pgbench txns | rows output | before | after | diff |
---|---|---|---|---|
100,000 | 1,200,057 | 15.219s | 12.058s | -20.8% |
200,000 | 1,900,073 | 23.849s | 18.227s | -23.6% |
300,000 | 2,600,083 | 31.222s | 24.182s | -22.6% |
400,000 | 3,300,101 | 39.237s | 31.146s | -20.7% |
500,000 | 4,000,111 | 47.055s | 36.420s | -22.7% |
@eulerto: this looks interesting. Thoughts?