rust-synapse-compress-state
rust-synapse-compress-state copied to clipboard
matrix-org
Incremental / partial compression
Right now, compressing a room requires holding its entire state in RAM, which can become arbitrarily large.
I don't know the internals of the compression algorithm, but given that state groups seem to be replaced one by one in an idempotent fashion, would it be possible to compress a room only partially somehow? Perhaps only the first N percent of rows, and do it a couple of times? Or maybe have multiple iterations of load, compress, free?
That would be tremendously helpful :+1:
We've successfully run this on some huge rooms. Is it actually a problem in practice?
Phew. I have to admit, I don't really remember the specifics back from February :|
Assuming a homeserver that isn't massively overprovisioned, extraordinarily allocating a couple hundred megs or even some gigs of RAM to a process for several hours is indeed not simple. I remember having to run this on another machine for a couple of my largest rooms that reason.
Ah yes, running it on a separate machine is probably a good idea. You can always tunnel the postgres connection over SSH.
I'm trying to compress a room with 430m state records which runs out of memory when run from a machine with 32GB memory (both the server and the machine from which this runs have 32GB) so I would argue that it is still a case.
The other side of this is that I don't really care how long it takes to run (well, almost) as long as it does.
I have room !GibBpYxFGNraRsZOyl:matrix.org (Techlore - Main) with 60+ millions of state groups (exact count is 64182319), and afraid to run full compression, because it will eat all my RAM :(
I have a room with 2 G (2248040941) rows. My estimate is that this will require 200 GB of RAM.
