Explore options on zstd for compression performance

[gtkramer]

Either from a CPU or memory perspective, let's explore if using the options for zstd affects compression performance to see if it can get better than xz by a degree enough to switch to zstd by default.

[ashleshaAtrey]

I explored various flag of zstd like fast, adapt, ultra and tried different levels. The size vs time trade off is not worth to switch zstd as default .

[bryteise]

@ashleshaAtrey I'm interested in if you still have those numbers around. It might be something that is worth exposing as a configuration option to the user. Some may find the trade-off to be worth it depending on the particular content that they are providing as eventually this would be used by 3rd-party bundles as well.

[rchiossi]

@bryteise It is already exposed. You can use it by setting COMPRESSION = ["external-zstd"] in builder.conf

[ashleshaAtrey]

Total size before compression 43184317440 Zstd Total size after compression 16528392879 CREATE FULLFILES 13m33.7s

zstd --fast=22 Total size after compression 25457700882 CREATE FULLFILES 13m19.447s

Zstd --adapt Total size after compression 16354637680 CREATE FULLFILES 13m25.032s

Zstd --fast Total size after compression 19045817625 CREATE FULLFILES 13m11.38s

Zstd --ultra Total size after compression 16380224911 CREATE FULLFILES 13m22.015s

[rchiossi]

Do you have the numbers for the other compression methods as well (xz, bzip2, gzip)?

[ashleshaAtrey]

Xz: Total size before compression 43183234560 Total size after compression 13877547116 CREATE FULLFILES 14m55.218s

bzip2 Total size before compression 43183935488 Total size after compression 16014229591 CREATE FULLFILES 12m18.674s

gzip Total size before compression 43182304256 Total size after compression 16910068787 CREATE FULLFILES 11m54.421s

[phmccarty]

I think that decompression speed is also worth cross-comparing, because xz tends to be slower than zstd in this area, and swupd has to decompress many files in course of its operation.

[phmccarty]

Also note Arjan's old blog post where he conducted a detailed cross-comparison for compression types. It would be awesome to produce a followup to that post with the latest findings.

[ashleshaAtrey]

I ran decompression tests on 906752 fullfiles using the tar utility,

For XZ compression, time took to decompress: 11134 seconds For Zstd compression, time took to decompress: 19767 seconds To decompress optimal size( mix of XZ or Zstd files): 12991 seconds

Next, I will work on finding the stats for memory used while compressing and decompressing those files.

[ashleshaAtrey]

Memory used while creating fullfiles: Alloc is bytes of allocated heap objects. TotalAlloc increases as heap objects are allocated, but unlike Alloc and HeapAlloc, it does not decrease when objects are freed Sys is the total bytes of memory obtained from the OS

Zstd compression Alloc = 6493 MiB TotalAlloc = 49925 MiB Sys = 9213 MiB
xz compression Alloc = 5850 MiB TotalAlloc = 49926 MiB Sys = 9344 MiB

[bryteise]

Interesting zstd is taking longer to decompress (and by a fairly significant amount too). That's really surprising.

[reaganlo]

@ashleshaAtrey Can you review if the numbers and units look correct.

	compression time (minutes)	decompression time (minutes)	memory usage (MiB)	compression size (Bytes)
xz	14.91666667	185.5666667	9344	13877547116
zstd	13.55	329.45	9213	16528392879

[phmccarty]

I am very surprised to see zstd have such a large decompression time. In my experience, decompression time for zstd has been dramatically faster than xz...

[bryteise]

Thinking about this a little more.

I wonder if there is a set of files which take disproportionately longer to decompress (but there are few of them) or files that take slightly longer to decompress (but there are many of them)?

Figuring this out would be enlightening. Do you have file by file time differences?

[ashleshaAtrey]

I will work on finding file by file time difference, dont have those stats handy.