openstreetmap-ng
openstreetmap-ng copied to clipboard
Trace file background recompression
Currently trace files are stored with the lightest zstd compression. We need an additional async background task, that in a separate thread will use heavy zstd compression and update the previous file. You can see how it's done in MailService so it doesn't delay the response.
The goal is to:
- In TraceService.upload, just before returning the trace, start an async task for compressing file_bytes.
- After heavier compression is achieved (level=22), update the Trace.file_id to point to the new file and delete the old file
Quoting from python-zstandard...
The C extension releases the GIL during non-trivial calls into the zstd C API. Non-trivial calls are notably compression and decompression. Trivial calls are things like parsing frame parameters.
...this may be enough to achieve parallelism:
loop = get_running_loop()
await loop.run_in_executor(None, compress_function, file_bytes)