aptly icon indicating copy to clipboard operation
aptly copied to clipboard

Publish could be faster by parallelizing uncompress threads

Open Hybrid512 opened this issue 2 years ago • 0 comments

Aptly is great but publishing snapshots can take ages with big mirrors when generating metadata. Adding more threading capacity (by adding more CPU cores for example) doesn't help by much since uncompress is done apparently in a single thread and this is especially slower with newer repositories based on zstd compression which takes a lot longer to uncompress than tar.gz.

image

Switching to more uncompressing threads (like N-1 threads for N vCPUs so that we still keep 1 vCPU for the system responsiveness) would probably greatly boost publish speed.

Your Environment

Aptly master branch : aptly version: '1.4.0+179+g0a179886' OS : Ubuntu 20.04 amd64 Storage: Filesystem (ext4) HW specs : KVM VM, host-passthrough CPU model (same as host, Intel Xeon Gold 6242 CPU @ 2.80GHz) 8 vCPUs, 4Gb RAM, 500GB disk Mirrors : ubuntu 20.04 and 22.04 (main, updates, security with main,restricted,universe,multiverse components)

Hybrid512 avatar May 16 '22 11:05 Hybrid512