citus
citus copied to clipboard
Enable binary logical replication for shard moves
Using binary encoding can save a lot of CPU cycles, both on the sender and on the receiver. Since the walsender and walreceiver processes are single threaded, this can matter a lot for the throughput if they are bottlenecked on CPU.
This feature is only available in PG14, not PG13. It should be safe to always enable because it's only used for types that support binary encoding according to the PG docs:
Even when this option is enabled, only data types that have binary send and receive functions will be transferred in binary.
But in case it causes problems, it can still be disabled by setting
citus.enable_binary_protocol
to false
.
Neat!
Do you have any data points to share, like with this schema & data size, each shard move time decreased from X -> Y on your local machine? Asking mostly for curiosity and some for future reference.
Even when this option is enabled, only data types that have binary send and receive functions will be transferred in binary.
Also, sounds like a good idea to add a single test -- if we do not have any -- just in case Citus hits some weird edge case of its own. regarding such types?
Also, sounds like a good idea to add a single test -- if we do not have any -- just in case Citus hits some weird edge case of its own. regarding such types?
Added some tests
Do you have any data points to share, like with this schema & data size, each shard move time decreased from X -> Y on your local machine?
Sadly I don't have any data points to share, but binary copy is significantly faster than text copy. I'm expecting this to at least perform just as well as text based logical replication. I planning to do some actual benchmarks a little bit before our release testing. But since setting up the benchmark environment is a bit time consuming I want to be able to test all the planned perf improvements for this release in one go (although I do want to test them separately).