adoptium
Post test job artifact gzip/pigz takes a very long time on some arch's
aqa-tests end of job “gzip & pigz” compression, currently they are both done and both use -9, the highest and slowest compression, this quite frequently takes 30mins+ on some archictures (eg. windows, pLinux). I've seen pLinux take over 1 hour before!
Has it been considered going for a half-way-house, say “-5" ?
Can you give some examples of jobs? My concern would be if it's compressing stuff for half an hour that's going to be quite a large artifact and so potentially indicative of other problems.
The example I found the other day: https://adoptium.slack.com/archives/C5219G28G/p1758614595564969?thread_ts=1758614530.576229&cid=C5219G28G was an aarch64 windows, and it took 20mins for 1.4Gb
This was it https://ci.adoptium.net/view/Test_openjdk/job/Test_openjdk25_hs_extended.openjdk_aarch64_windows/19/console
Probably the disks on these nodes are just rubbish!!
Probably the disks on these nodes are just rubbish!!
That seems the most likely option for a desktop windows system IMHO. There are over 150k files in that archive. I seriously doubt the compression is the limiting factor here 🤷🏻
Quick test on my (pre-COVID) aarch64 windows laptop (Snapdragon 850 CPU) using the tarball from that job with gzip (not pigz) under WSL rather than cygwin:
- Compressing the tarball with the default options: 34s
- Compressing the tarball with
gzip -9: 83s - Creating and compressing the tar.gz from the extracted files with
-9: 57m (thegzipprocess was barely using 2% of the CPU during that - less that some of the Windows Defender processes which were ~10%) - Creating and compressing the tar.gz from the extracted files on my 7-year old Linux laptop with
gzip -9: 31s