Please allow tweaking the SquashFS compression

[l3u]

Hi devs,

first of all thanks a lot for this fine piece of software, which seems to be the only viable successor of linuxdeploy-qt :-)

I searched for ways to generate the smallest possible AppImage for a Qt application. I use a static build, which greatly reduces the size, but leaving out features doesn't help much anymore.

So I examined if the compression method used to create the AppImage itself could be tweaked. Zstd is used to compress the data. SquashFS seems to not only support zstd, but also other compression algorithms – above all LZMA.

However, linuxdeploy uses an embedded version of mksquashfs, which only supports zstd. I found out that one could pass other compression algorithms to be used via the LDAI_COMP env variable – but as the packaged mksquashfs only supports zstd, that is not really meaningful.

I did a simple test to check out if LZMA was worth it. I mounted a test AppImage and tared the contents. The uncompressed tarball is 17.0M. Compressing it with zstd results in 7.3M, almost the size of the AppImage itself (7.5M). However, compressing it using xz, the tarball only takes 5.7M – that is almost 1/4 less!

While xz takes way longer to compress the data compared to zstd (5.6s vs. 0.1s), decompression is way faster. Admittedly, it's still slower for xz compared with zstd (0.2s vs. less than 0.1s), but considering the significant smaller AppImage, it seems to be worth it – as least as an option.

So – it would be really nice if one could choose to either use LZMA or zstd when creating an AppImage, preferably with the possibility to tweak the compression settings to be able to create smaller AppImages.

Cheers, Tobias

[CJYate]

This issue may be preventing my appimages being run.

Running the image:

QSocketNotifier: Can only be used with threads started with QThread
Squashfs image uses (null) compression, this version supports only xz, zlib.
ERROR: appimage_shall_not_be_integrated : sqfs_open_image error: /home/multirig/Desktop/Flair_7.7.0_Alpha.AppImage
AppImageLauncher error: appimage_shall_not_be_integrated() failed (returned -1)
Squashfs image uses (null) compression, this version supports only xz, zlib.
ERROR: appimage_is_terminal_app : sqfs_open_image error: /home/multirig/Desktop/Flair_7.7.0_Alpha.AppImage
AppImageLauncher error: appimage_is_terminal_app() failed (returned -1)
fuse: memory allocation failed
squashfuse 0.5.2 (c) 2012 Dave Vasilevsky

Usage: squashfuse [options] ARCHIVE MOUNTPOINT

(null) options:
    -o offset=N            offset N bytes into ARCHIVE to mount
    -o subdir=PATH         mount subdirectory PATH of ARCHIVE
    -o notify_pipe=PATH    named pipe that will receive 's' (success)
                           or 'f' (failure) when the mountpoint is ready
    -o timeout=N           idle N seconds for automatic unmount
    -o uid=N               set file owner to uid N
    -o gid=N               set file group to gid N

FUSE options:
    -h   --help            print help
    -V   --version         print version
    -d   -o debug          enable debug output (implies -f)
    -f                     foreground operation
    -s                     disable multi-threaded operation
    -o clone_fd            use separate fuse device fd for each thread
                           (may improve performance)
    -o max_idle_threads    the maximum number of idle worker threads
                           allowed (default: -1)
    -o max_threads         the maximum number of worker threads
                           allowed (default: 10)
fuse: memory allocation failed
squashfuse 0.5.2 (c) 2012 Dave Vasilevsky

Usage: squashfuse [options] ARCHIVE MOUNTPOINT

(null) options:
    -o offset=N            offset N bytes into ARCHIVE to mount
    -o subdir=PATH         mount subdirectory PATH of ARCHIVE
    -o notify_pipe=PATH    named pipe that will receive 's' (success)
                           or 'f' (failure) when the mountpoint is ready
    -o timeout=N           idle N seconds for automatic unmount
    -o uid=N               set file owner to uid N
    -o gid=N               set file group to gid N

FUSE options:
    -h   --help            print help
    -V   --version         print version
    -d   -o debug          enable debug output (implies -f)
    -f                     foreground operation
    -s                     disable multi-threaded operation
    -o clone_fd            use separate fuse device fd for each thread
                           (may improve performance)
    -o max_idle_threads    the maximum number of idle worker threads
                           allowed (default: -1)
    -o max_threads         the maximum number of worker threads
                           allowed (default: 10)
squashfuse 0.5.2 (c) 2012 Dave Vasilevsky

Usage: squashfuse [options] ARCHIVE MOUNTPOINT

(null) options:
    -o offset=N            offset N bytes into ARCHIVE to mount
    -o subdir=PATH         mount subdirectory PATH of ARCHIVE
    -o notify_pipe=PATH    named pipe that will receive 's' (success)
                           or 'f' (failure) when the mountpoint is ready
    -o timeout=N           idle N seconds for automatic unmount
    -o uid=N               set file owner to uid N
    -o gid=N               set file group to gid N

FUSE options:
    -h   --help            print help
    -V   --version         print version
    -d   -o debug          enable debug output (implies -f)
    -f                     foreground operation
    -s                     disable multi-threaded operation
    -o clone_fd            use separate fuse device fd for each thread
                           (may improve performance)
    -o max_idle_threads    the maximum number of idle worker threads
                           allowed (default: -1)
    -o max_threads         the maximum number of worker threads
                           allowed (default: 10)
Can't open squashfs image: Bad address

Cannot mount AppImage, please check your FUSE setup.
You might still be able to extract the contents of this AppImage 
if you run it with the --appimage-extract option. 
See https://github.com/AppImage/AppImageKit/wiki/FUSE 
for more information
execv error: No such file or directory

on --appimage-extract:

QSocketNotifier: Can only be used with threads started with QThread
This doesn't look like a squashfs image.
Failed to open squashfs image

This does extract perfectly well on the machine I used to create the package, not on a system I copy it to.

[TheAssassin]

That's got nothing to do with the AppImage, but with your outdated version of AppImageLauncher. Please install the latest 3.0 alpha (yes, despite the name, it's good for daily use).

[TheAssassin]

@l3u Zstd is known for its good decompression performance, so I'm wondering how xz could be significantly faster. That said, I think 0.2s isn't a bad startup time. Zstd is also used by default nowadays by the AppImage plugin (to which I will transfer this issue soon).

In the end, the AppImage plugin is just a wrapper around appimagetool. You can configure the compression with the LDAI_COMP environment variable, provided that appimagetool supports it. See https://github.com/linuxdeploy/linuxdeploy-plugin-appimage.

[TheAssassin]

I apologize, I didn't read your (very lengthy) statement correctly. It states correctly that the flag exists. But you are requesting xz (LZMA) support. I think that was removed by appimagetool a while ago because Zstd provides the best trade-off of all compression algorithms. I'll find you a link.

[TheAssassin]

See https://github.com/AppImage/appimagetool/issues/69. There it's also mentioned that mksquashfs is built without xz support and bundled with appimagetool. Please open an issue at https://github.com/AppImage/appimagetool/.

Edit: by the way, thanks for the kind words!

[l3u]

Thanks for the insight! Well, if LZMA support was there and has been removed intentionally – I fear requesting to add it again won't be very promising, will it?!

[TheAssassin]

I am sure it would be considered (given the fact that I'll be one of the reviewers of such an issue). It just needs to be well founded.

Currently, we are investigating DwarFS as an alternative to squashfs.

Edit: xz/lzma used to be the default (instead of the also-supported zlib/gzip).

[l3u]

Okay, then I'll file an issue about that.

I didn't knew DwarFS until now, thanks for the info! This is interesting! I just tested the latest pre-built mkdwarfs:

My final, zstd-compressed AppImage is 8.9M. When I unpack it, the squashfs-root is 22.0M. When I create a DwarFS from that, it shrinks to 6.5M. Plus some overhead for the AppImage stuff of course, but it definitely would be way smaller than the zstd version.

Would be nice if we could achieve better compression. The best thing would be if we simply had a choice here. Of course, zstd is really fast for what it compresses. But for my use-case, size would be more important than decompression speed, esp. when we're talking about split seconds.

[l3u]

Here we are :-)

[l3u]

… aaand it's already closed :-(

[CJYate]

That's got nothing to do with the AppImage, but with your outdated version of AppImageLauncher. Please install the latest 3.0 alpha (yes, despite the name, it's good for daily use).

Massively helpful response, thank you!