AppImageSpec icon indicating copy to clipboard operation
AppImageSpec copied to clipboard
verified publisher icon AppImage

Systematically benchmark compression algorithm, compression factor, block size

Open probonopd opened this issue 1 year ago • 10 comments

Execute a systematic and reproducible benchmark to find the optimal combination of

  • Compression algorithm
  • Compression factor
  • Block size

in terms of

  • File size
  • Applicartion startup time
  • zsync efficiency (delta update size)

for

  • zlib
  • lzma
  • zstandard
  • dwarfs
  • libdeflate

References:

  • https://github.com/AppImage/appimagetool/issues/64
  • https://github.com/AppImage/AppImageSpec/issues/36

probonopd avatar Aug 08 '24 02:08 probonopd

This test matrix includes 5 compression algorithms (zlib, lzma, zstandard, dwarfs, and libdeflate), 3 compression factors (low, mid, high), and 3 block sizes (256KB, 512KB, and 1MB), resulting in a total of 45 test cases.

Compression Algorithm Compression Factor Block Size File Size Application Startup Time zsync Efficiency (Delta Update Size)
squashfs with zlib 6 256KB
squashfs with zlib 6 512KB
squashfs with zlib 6 1MB
squashfs with zlib 9 256KB
squashfs with zlib 9 512KB
squashfs with zlib 9 1MB
squashfs with lzma 0 (fast) 256KB
squashfs with lzma 0 (fast) 512KB
squashfs with lzma 0 (fast) 1MB
squashfs with lzma 6 (normal) 256KB
squashfs with lzma 6 (normal) 512KB
squashfs with lzma 6 (normal) 1MB
squashfs with lzma 9 (ultra) 256KB
squashfs with lzma 9 (ultra) 512KB
squashfs with lzma 9 (ultra) 1MB
squashfs with zstandard 3 256KB
squashfs with zstandard 3 512KB
squashfs with zstandard 3 1MB
squashfs with zstandard 10 256KB
squashfs with zstandard 10 512KB
squashfs with zstandard 10 1MB
squashfs with zstandard 19 256KB
squashfs with zstandard 19 512KB
squashfs with zstandard 19 1MB
dwarfs 128 256KB
dwarfs 128 512KB
dwarfs 128 1MB
dwarfs 256 256KB
dwarfs 256 512KB
dwarfs 256 1MB
libdeflate 6 256KB
libdeflate 6 512KB
libdeflate 6 1MB
libdeflate 9 256KB
libdeflate 9 512KB
libdeflate 9 1MB

probonopd avatar Aug 08 '24 02:08 probonopd

Volunteers?

probonopd avatar Aug 08 '24 02:08 probonopd

Something that needs to be considered as well is to take into account how long it normally takes for the same application not as an appimage to start.

For example we might see that on a very big application a certain algo is 30% faster, but that application even when not being an appimage due to its size takes several seconds to start anyway, and that 30% ends up being a very small percentage of the overall delay for the app.

Same way for very small applications, the speed difference might not matter much, because they are very small and take no time regardless.

Where problems can happen is with the mid size applications that you normally expect to start fast, those are web-browsers in other words.

Right now zstd with the current default block size is actually very good, I will do some benchmarks comparing the size and startup times, however I can't measure zsync efficiency since that seems quite a bit more work.

I'm also interested to know how this affects very old hardware, IE, some pre sandy bridge cpu for example, my hardware is from 2016 and not the worst I would say lol.

Samueru-sama avatar Aug 08 '24 05:08 Samueru-sama

Application startup times also depend on the hardware. On systems with slow disk but fast CPU, a highly compressed image may lead to faster application launch times than uncompressed files (iirc, I have seen this myself with a large application in the past, likely with a spinning drive). It always depends where the performance bottleneck is on a particular system.

So for this to be really scientific, we'd have to execute the test matrix for typical defined machines.

But then, we are not exactly writing a dissertation here ;-)

probonopd avatar Aug 09 '24 08:08 probonopd

I'm probably a bit late, but I think EROFS is an interesting option as well.

  • wide kernel support (since 5.4, Ubuntu 20.04, Alpine since 3.11)
  • fuse support,
  • compression:
    • zstd,
    • lzma,
    • lz4

dwarfs is incredibly performant, but gpl3 makes it impossible to use with proprietary programs packaged in appimage. At the same time, AppImage packaging is used by large companies, so the introduction of dwarfs will cut off a significant part of the audience

Drsheppard01 avatar Sep 30 '24 01:09 Drsheppard01

I got these results running on a VM with 12 GiB Memory and 4 cores: results.csv

Compression Method Compression Parameters Block Size Compression Time FS Size Startup Time (average of 5 runs) docx to pdf Time (average of 5 runs) zsync download size
SquashFS gzip -Xcompression-level 1 16k 18110.2 ms 1119.14 MiB 1810.8 ms 3494.1 ms 168.85 MiB
SquashFS gzip -Xcompression-level 1 64k 16387.5 ms 930.97 MiB 1781.4 ms 3579.7 ms 171.56 MiB
SquashFS gzip -Xcompression-level 1 256k 15713.5 ms 892.99 MiB 1942.4 ms 3737.5 ms 207.79 MiB
SquashFS gzip -Xcompression-level 1 512k 15696.9 ms 886.75 MiB 1942.3 ms 3743.1 ms 227.78 MiB
SquashFS gzip -Xcompression-level 1 1M 15614.7 ms 883.83 MiB 2047.6 ms 3991.9 ms 262.65 MiB
SquashFS gzip -Xcompression-level 6 16k 24765.2 ms 1017.00 MiB 1849.4 ms 3423.8 ms 159.67 MiB
SquashFS gzip -Xcompression-level 6 64k 25149.3 ms 803.89 MiB 1795.3 ms 3436.5 ms 159.65 MiB
SquashFS gzip -Xcompression-level 6 256k 26578.8 ms 745.39 MiB 1777.4 ms 3512.6 ms 190.23 MiB
SquashFS gzip -Xcompression-level 6 512k 27269.3 ms 735.77 MiB 1840.6 ms 3662.1 ms 207.10 MiB
SquashFS gzip -Xcompression-level 6 1M 26230.7 ms 731.28 MiB 1950.7 ms 3830.9 ms 236.83 MiB
SquashFS gzip -Xcompression-level 9 16k 31317.2 ms 1013.75 MiB 1769.3 ms 3371.3 ms 159.16 MiB
SquashFS gzip -Xcompression-level 9 64k 42813.5 ms 798.62 MiB 1733.3 ms 3434.9 ms 159.42 MiB
SquashFS gzip -Xcompression-level 9 256k 48116.2 ms 739.41 MiB 1778.2 ms 3483.0 ms 188.72 MiB
SquashFS gzip -Xcompression-level 9 512k 49885.5 ms 729.69 MiB 1873.3 ms 3685.6 ms 206.14 MiB
SquashFS gzip -Xcompression-level 9 1M 48892.3 ms 725.17 MiB 1955.9 ms 3940.3 ms 235.60 MiB
SquashFS lzo -Xcompression-level 1 16k 21924.3 ms 1267.65 MiB 1534.1 ms 3096.9 ms 169.46 MiB
SquashFS lzo -Xcompression-level 1 64k 20306.3 ms 1012.04 MiB 1577.4 ms 3065.8 ms 159.91 MiB
SquashFS lzo -Xcompression-level 1 256k 19828.8 ms 940.37 MiB 1592.8 ms 3145.6 ms 152.45 MiB
SquashFS lzo -Xcompression-level 1 512k 20631.9 ms 928.72 MiB 1619.6 ms 3172.9 ms 146.83 MiB
SquashFS lzo -Xcompression-level 1 1M 20098.3 ms 923.09 MiB 1755.0 ms 3458.9 ms 142.23 MiB
SquashFS lzo -Xcompression-level 6 16k 27793.1 ms 1192.38 MiB 1510.0 ms 2987.9 ms 159.45 MiB
SquashFS lzo -Xcompression-level 6 64k 29080.7 ms 915.13 MiB 1544.1 ms 3012.1 ms 146.54 MiB
SquashFS lzo -Xcompression-level 6 256k 31329.9 ms 833.88 MiB 1567.2 ms 3074.6 ms 136.98 MiB
SquashFS lzo -Xcompression-level 6 512k 32028.0 ms 820.70 MiB 1585.2 ms 3182.4 ms 130.70 MiB
SquashFS lzo -Xcompression-level 6 1M 31352.5 ms 814.33 MiB 1691.9 ms 3361.3 ms 126.41 MiB
SquashFS lzo -Xcompression-level 9 16k 50543.6 ms 1183.98 MiB 1570.2 ms 3069.0 ms 158.34 MiB
SquashFS lzo -Xcompression-level 9 64k 67672.1 ms 902.67 MiB 1556.2 ms 3041.0 ms 144.56 MiB
SquashFS lzo -Xcompression-level 9 256k 79792.8 ms 819.64 MiB 1534.2 ms 3039.5 ms 134.66 MiB
SquashFS lzo -Xcompression-level 9 512k 83141.9 ms 806.20 MiB 1602.6 ms 3198.3 ms 128.41 MiB
SquashFS lzo -Xcompression-level 9 1M 81887.2 ms 799.70 MiB 1651.4 ms 3217.6 ms 124.16 MiB
SquashFS lz4   16k 12959.3 ms 1472.75 MiB 1362.8 ms 2783.3 ms 211.53 MiB
SquashFS lz4   64k 11803.8 ms 1165.39 MiB 1385.0 ms 2751.1 ms 210.94 MiB
SquashFS lz4   256k 11077.0 ms 1092.33 MiB 1445.8 ms 2792.0 ms 224.16 MiB
SquashFS lz4   512k 10743.2 ms 1080.90 MiB 1423.5 ms 2791.9 ms 224.01 MiB
SquashFS lz4   1M 10781.8 ms 1075.60 MiB 1419.8 ms 2815.3 ms 220.75 MiB
SquashFS xz -Xdict-size 100% 16k 157594.6 ms 962.06 MiB 2672.6 ms 4816.1 ms 167.56 MiB
SquashFS xz -Xdict-size 100% 64k 142406.6 ms 708.87 MiB 2704.6 ms 5130.0 ms 173.12 MiB
SquashFS xz -Xdict-size 100% 256k 149722.1 ms 604.63 MiB 2615.8 ms 5099.6 ms 187.62 MiB
SquashFS xz -Xdict-size 100% 512k 155167.6 ms 573.89 MiB 2811.0 ms 5681.2 ms 195.02 MiB
SquashFS xz -Xdict-size 100% 1M 175004.0 ms 549.29 MiB 2983.1 ms 5923.1 ms 210.65 MiB
SquashFS zstd -Xcompression-level 1 16k 13508.4 ms 1079.95 MiB 1460.5 ms 2922.0 ms 167.31 MiB
SquashFS zstd -Xcompression-level 1 64k 12735.8 ms 863.07 MiB 1484.1 ms 2968.1 ms 170.44 MiB
SquashFS zstd -Xcompression-level 1 256k 11946.1 ms 771.79 MiB 1527.9 ms 2962.3 ms 182.90 MiB
SquashFS zstd -Xcompression-level 1 512k 11370.3 ms 761.88 MiB 1463.8 ms 2896.1 ms 194.40 MiB
SquashFS zstd -Xcompression-level 1 1M 10859.0 ms 751.70 MiB 1522.0 ms 3112.5 ms 213.06 MiB
SquashFS zstd -Xcompression-level 6 16k 20443.0 ms 1009.18 MiB 1547.2 ms 3005.2 ms 150.32 MiB
SquashFS zstd -Xcompression-level 6 64k 17160.3 ms 786.94 MiB 1529.5 ms 3104.2 ms 146.00 MiB
SquashFS zstd -Xcompression-level 6 256k 16392.8 ms 695.48 MiB 1577.8 ms 3073.1 ms 165.67 MiB
SquashFS zstd -Xcompression-level 6 512k 16497.4 ms 666.01 MiB 1559.6 ms 3123.8 ms 176.20 MiB
SquashFS zstd -Xcompression-level 6 1M 15757.7 ms 642.93 MiB 1582.3 ms 3094.2 ms 192.88 MiB
SquashFS zstd -Xcompression-level 11 16k 56160.3 ms 1005.07 MiB 1542.4 ms 3037.8 ms 149.72 MiB
SquashFS zstd -Xcompression-level 11 64k 38756.4 ms 773.32 MiB 1496.7 ms 2955.2 ms 142.71 MiB
SquashFS zstd -Xcompression-level 11 256k 47508.0 ms 674.18 MiB 1597.6 ms 2994.8 ms 161.33 MiB
SquashFS zstd -Xcompression-level 11 512k 23702.5 ms 651.57 MiB 1595.3 ms 2999.3 ms 174.39 MiB
SquashFS zstd -Xcompression-level 11 1M 24029.9 ms 626.26 MiB 1530.3 ms 3077.0 ms 189.20 MiB
SquashFS zstd -Xcompression-level 15 16k 159752.1 ms 975.35 MiB 1596.3 ms 3044.4 ms 143.35 MiB
SquashFS zstd -Xcompression-level 15 64k 92158.8 ms 744.79 MiB 1587.4 ms 3090.8 ms 142.75 MiB
SquashFS zstd -Xcompression-level 15 256k 102846.7 ms 638.89 MiB 1586.0 ms 3145.9 ms 165.22 MiB
SquashFS zstd -Xcompression-level 15 512k 59426.8 ms 646.09 MiB 1531.4 ms 3066.1 ms 169.68 MiB
SquashFS zstd -Xcompression-level 15 1M 70819.5 ms 620.51 MiB 1582.0 ms 3139.0 ms 184.97 MiB
SquashFS zstd -Xcompression-level 19 16k 306681.6 ms 971.67 MiB 1573.4 ms 2985.6 ms 140.86 MiB
SquashFS zstd -Xcompression-level 19 64k 337880.4 ms 738.67 MiB 1489.3 ms 2988.1 ms 137.11 MiB
SquashFS zstd -Xcompression-level 19 256k 293177.2 ms 633.54 MiB 1513.3 ms 3009.7 ms 160.65 MiB
SquashFS zstd -Xcompression-level 19 512k 257798.8 ms 601.88 MiB 1537.7 ms 3064.6 ms 170.96 MiB
SquashFS zstd -Xcompression-level 19 1M 250558.4 ms 575.98 MiB 1571.6 ms 3140.7 ms 189.89 MiB
SquashFS zstd -Xcompression-level 22 16k 357046.5 ms 971.59 MiB 1523.3 ms 3011.1 ms 140.85 MiB
SquashFS zstd -Xcompression-level 22 64k 451164.6 ms 738.42 MiB 1478.1 ms 2979.2 ms 137.81 MiB
SquashFS zstd -Xcompression-level 22 256k 395652.6 ms 633.24 MiB 1544.2 ms 3036.7 ms 160.58 MiB
SquashFS zstd -Xcompression-level 22 512k 353558.6 ms 601.49 MiB 1528.2 ms 3072.5 ms 171.46 MiB
SquashFS zstd -Xcompression-level 22 1M 342044.9 ms 575.53 MiB 1566.8 ms 3127.2 ms 189.75 MiB
My script 
import itertools
import os
import pathlib
import re
import shlex
import subprocess
import time

file_ver1 = "./LibreOffice-24.8.3.2.full.help-x86_64.AppImage"
file_ver2 = "./LibreOffice-24.8.4.2.full.help-x86_64.AppImage"

block_sizes = ["16k", "64k", "256k", "512k", "1M"]
comps_squashfs = {
    "gzip": {"-Xcompression-level": ["1", "6", "9"]},
    "lzo": {
        "-Xcompression-level": ["1", "6", "9"],
    },
    "lz4": {},
    "xz": {
        "-Xdict-size": ["100%"],
    },
    "zstd": {
        "-Xcompression-level": ["1", "6", "11", "15", "19", "22"],
    },
}

output = open("results.csv", "w")
output.write(
    "Compression Method,Compression Parameters,Block Size,Compression Time,"
    "FS Size,Startup Time (average of 5 runs),docx to pdf Time (average of 5 runs),zsync download size"
)
output.flush()

try:
    os.mkdir("zsyncout")
except FileExistsError:
    pass

if not os.path.exists("ver1.AppDir"):
    subprocess.Popen([file_ver1, "--appimage-extract"]).wait()
    os.rename("squashfs-root", "ver1.AppDir")
if not os.path.exists("ver2.AppDir"):
    subprocess.Popen([file_ver2, "--appimage-extract"]).wait()
    os.rename("squashfs-root", "ver2.AppDir")

with open("uruntime-appimage-x86_64", "rb") as f:
    runtime = f.read()

for comp in comps_squashfs:
    # product all possible combinations of compression parameters
    comp_params = list(
        itertools.product(*list(list([i, j] for j in comps_squashfs[comp][i]) for i in comps_squashfs[comp]))
    )
    for comp_param in comp_params:
        for block_size in block_sizes:
            output.write("\n")
            output.write(f"SquashFS {comp},{shlex.join(sum(comp_param, []))},{block_size},")
            output.flush()

            print(
                "Preparing AppImages for compression method %s with parameters %s and block size %s"
                % (comp, sum(comp_param, []), block_size)
            )
            # Prepare the AppImages
            command = ["mksquashfs", "ver1.AppDir", "file_ver1.squashfs", "-root-owned", "-noappend", "-comp", comp]
            command.extend(["-b", block_size])
            for i in comp_param:
                command.extend(i)
            # file_ver1 is only the base file, so do not measure efficiency
            p = subprocess.Popen(command)
            if p.wait() != 0:
                print("Error in compression")
                continue
            with open("file_ver1.squashfs", "rb") as f:
                squashfs = f.read()
            os.remove("file_ver1.squashfs")
            with open("file_ver1.AppImage", "wb") as f:
                f.write(runtime)
                f.write(squashfs)
            del squashfs
            try:
                os.chmod("file_ver1.AppImage", 0o755)
            except PermissionError:
                pass

            # Compression time
            command[1] = "ver2.AppDir"
            command[2] = "file_ver2.squashfs"
            start = time.time()
            p = subprocess.Popen(command)
            if p.wait() != 0:
                print("Error in compression")
                continue
            output.write(f"{(time.time() - start) * 1000:.1f}ms,")
            output.flush()
            with open("file_ver2.squashfs", "rb") as f:
                squashfs = f.read()
            os.remove("file_ver2.squashfs")
            with open("file_ver2.AppImage", "wb") as f:
                f.write(runtime)
                f.write(squashfs)
            output.write(f"{len(squashfs)},")
            output.flush()
            print("SquashFS size: %d" % len(squashfs))
            del squashfs
            try:
                os.chmod("file_ver2.AppImage", 0o755)
            except PermissionError:
                pass

            # Startup time
            print("Measuring startup time, 10 rounds, only last 5 rounds count")
            for _ in range(2):
                startup_times = []
                for i in range(5):
                    start = time.time()
                    p = subprocess.Popen(["./file_ver2.AppImage", "--terminate_after_init"])
                    p.wait()
                    used_time = time.time() - start
                    startup_times.append(used_time)
                    print("Startup time round %d: %f" % (i + 1, used_time))
            output.write(f"{sum(startup_times) * 1000 / 5:.1f} ms,")
            output.flush()

            # docx to pdf time
            print("Measuring docx to pdf time, 10 rounds, only last 5 rounds count")
            for _ in range(2):
                docx_to_pdf_times = []
                for i in range(5):
                    start = time.time()
                    p = subprocess.Popen(
                        [
                            "./file_ver2.AppImage",
                            "--convert-to",
                            "pdf",
                            pathlib.Path("test.docx").absolute(),
                            "--outdir",
                            pathlib.Path(".").absolute(),
                        ]
                    )
                    p.wait()
                    used_time = time.time() - start
                    docx_to_pdf_times.append(used_time)
                    print("docx to pdf time round %d: %f" % (i + 1, used_time))
            output.write(f"{sum(docx_to_pdf_times) * 1000 / 5:.1f} ms,")
            output.flush()

            # zsync download size
            print("Creating zsync file")
            p = subprocess.Popen(["zsyncmake", "file_ver2.AppImage"])
            if p.wait() != 0:
                print("Error in zsync")
                continue
            print("Calculating zsync download size")
            p = subprocess.Popen(
                ["zsync", "-i", "file_ver1.AppImage", "file_ver2.AppImage.zsync", "-o", "./zsyncout"],
                stderr=subprocess.PIPE,
            )
            zsync_output = p.stderr.read().decode("utf-8")
            percentage = re.search(r"(\d+\.\d+)%", zsync_output).group(1)
            p.wait()
            try:
                os.remove("zsyncout.part")
            except FileNotFoundError:
                pass
            output.write(f"{(1 - float(percentage) / 100) * os.path.getsize('file_ver2.AppImage'):.0f}")
            output.flush()

output.close()

CarlGao4 avatar Jan 09 '25 00:01 CarlGao4

This is amazing @CarlGao4 thank you.

I modified the script to not test as many options and also dropped lzo all together, tested with the Brave AppImage:

Tested on actual hardware: Xeon E5-2640 V4 + 16 GiB of mem.

Compression Method Compression Parameters Block Size Compression Time FS Size Startup Time (average of 5 runs) zsync download size
SquashFS gzip -Xcompression-level 1 128k 696.1ms 185876480 3009.8 ms 143952033
SquashFS gzip -Xcompression-level 1 512k 712.8ms 184889344 5273.6 ms 145323560
SquashFS gzip -Xcompression-level 1 1M 899.6ms 184729600 6895.4 ms 146543394
SquashFS gzip -Xcompression-level 6 128k 1627.3ms 172199936 2964.2 ms 133914685
SquashFS gzip -Xcompression-level 6 512k 1701.2ms 170659840 4850.3 ms 135070067
SquashFS gzip -Xcompression-level 6 1M 2268.4ms 170401792 6764.2 ms 136114317
SquashFS gzip -Xcompression-level 9 128k 3142.5ms 171626496 2901.7 ms 133485752
SquashFS gzip -Xcompression-level 9 512k 3480.5ms 170024960 4779.2 ms 134410068
SquashFS gzip -Xcompression-level 9 1M 4406.3ms 169750528 6673.8 ms 135790727
SquashFS lz4   128k 262.4ms 231120896 1147.7 ms 170373122
SquashFS lz4   512k 233.2ms 229834752 1313.5 ms 166612261
SquashFS lz4   1M 1757.7ms 229613568 1495.2 ms 166220104
SquashFS xz -Xdict-size 100% 128k 10155.6ms 148647936 7873.7 ms 112099874
SquashFS xz -Xdict-size 100% 512k 11458.4ms 141467648 14832.4 ms 107812675
SquashFS xz -Xdict-size 100% 1M 14598.5ms 138706944 20413.6 ms 106533330
SquashFS zstd -Xcompression-level 1 128k 217.8ms 187547648 1512.8 ms 137036518
SquashFS zstd -Xcompression-level 1 512k 221.8ms 186511360 2072.7 ms 137079302
SquashFS zstd -Xcompression-level 1 1M 263.4ms 186064896 2559.2 ms 136955654
SquashFS zstd -Xcompression-level 15 128k 5820.6ms 158380032 1777.1 ms 115151666
SquashFS zstd -Xcompression-level 15 512k 3303.8ms 162115584 2154.9 ms 116065886
SquashFS zstd -Xcompression-level 15 1M 4653.0ms 158838784 2715.6 ms 113483386
SquashFS zstd -Xcompression-level 22 128k 17713.8ms 156803072 1808.3 ms 116179759
SquashFS zstd -Xcompression-level 22 512k 13204.2ms 149749760 2600.0 ms 111798229
SquashFS zstd -Xcompression-level 22 1M 15394.0ms 146882560 3256.2 ms 110212187

I don't know how to make the table look all good like you did here. 😅

EDIT: I had this hack running in the background to kill the brave window every time it opened:

while true; do
    window_class="$(xdotool getactivewindow getwindowclassname 2>/dev/null)"
    if echo "$window_class" | grep -q "Brave-browser"; then
        killall brave
    fi
    sleep 0.0001
done

Samueru-sama avatar Jan 09 '25 09:01 Samueru-sama

Oh, I opened the csv in Excel and just copy the table and paste it here, it will automatically form a table in markdown syntax Besides, I changed some display format before exporting it here

CarlGao4 avatar Jan 09 '25 09:01 CarlGao4

I'll try to test DwarFS later

CarlGao4 avatar Jan 09 '25 09:01 CarlGao4

I'll try to test DwarFS later

That's going to be a lot of block sizes btw, with dwarfs iirc it can be as high as 64MB. And the impact also varies greatly depending on the application that's tested.

With this AppImage of GIMP for example -S25(32MB) was good while with Cromite I had to lower it to -S20 (1MB) otherwise it takes too long for the browser to open.

Samueru-sama avatar Jan 09 '25 09:01 Samueru-sama