centraldogma icon indicating copy to clipboard operation
centraldogma copied to clipboard
verified publisher icon line

Compress replication commands when serializing

Open trustin opened this issue 5 years ago • 2 comments

When a user creates a large amount of big commits in a replicated environment, a replica might suffer from heap shortage, because we serialize commits (PushAsIsCommand to be precise) into JSON and store it into ZooKeeper.

To reduce the risk of running out of memory, we could compress the serialized replication command (zstd?)

trustin avatar Feb 08 '20 08:02 trustin

  • Allow to disable compression so that gradual rollout is possible.
  • Consider supporting multiple compression algorithms - 'Zstd' if JNI library is available. 'deflate' otherwise.
    • Think about the ways to expose compression options such as level.

trustin avatar Feb 08 '20 09:02 trustin

I'm interested in this issue.

Think about the ways to expose compression options such as level.

I think that we can provide an option like CompressionType that has ZSTD, and DEFLATE (and BROTLI?).

0x1306e6d avatar Dec 15 '21 15:12 0x1306e6d