openstreetmap-ng icon indicating copy to clipboard operation
openstreetmap-ng copied to clipboard

Trace file background recompression

Open Zaczero opened this issue 6 months ago • 0 comments

Currently trace files are stored with the lightest zstd compression. We need an additional async background task, that in a separate thread will use heavy zstd compression and update the previous file. You can see how it's done in MailService so it doesn't delay the response.

The goal is to:

  • In TraceService.upload, just before returning the trace, start an async task for compressing file_bytes.
  • After heavier compression is achieved (level=22), update the Trace.file_id to point to the new file and delete the old file

Quoting from python-zstandard...

The C extension releases the GIL during non-trivial calls into the zstd C API. Non-trivial calls are notably compression and decompression. Trivial calls are things like parsing frame parameters.

...this may be enough to achieve parallelism:

loop = get_running_loop()
await loop.run_in_executor(None, compress_function, file_bytes)

Zaczero avatar Jul 31 '24 12:07 Zaczero