Should compiled bytecode (__pycache__/*.pyc) be bundled with the AppImage?

[mxmlnkn]

Hello,

This is more of a question than an issue. Maybe someone here knows the answer.

Should compiled bytecode (.pyc) be bundled with the AppImage?

I have never heard of bugs caused by accidentally bundling bytecode or having some bytecode leftover after updates. I would assume that it is simply ignored or regenerated when there are incompatibilities. The glossary states:

This “intermediate language” is said to run on a virtual machine that executes the machine code corresponding to each bytecode. Do note that bytecodes are not expected to work between different Python virtual machines, nor to be stable between Python releases.

To me, this seems to imply that everything should be fine as long as the same Python binary is used, which is the case for an AppImage bundling it.

Some solutions such as PyInstaller seem to always bundle the bytecode and even offer to remove the original source code.

So, would it not make sense to bundle it to reduce startup times? Else, it might try to compile the .py files on each AppImage start.

Even if it would not make sense to bundle it, should the -B option be used for the Python interpreter so that it does not even try to write out .pyc files somewhere? As far as I know, the AppImage is mounted on a read-only file system, so it shouldn't matter much, except maybe some failed write syscalls. It may make a difference when running the extracted AppDir via AppRun, but I guess then it would be desired to create the __pycache__ / .pyc files. It might also not make sense when not using the isolated mode.

It seems that a normal pip install inside the AppImage already creates .pyc files, but my build script deleted them with find "$APP_DIR" -name '__pycache__' -print0 | xargs -0 rm -r to optimize the AppImage size, I suppose. But, I assume that this only creates .pyc files for actually imported modules. It seems inconsistent to have some modules compiled and some not. Ergo, should all the .pyc be deleted, or should I run compileall?

It seems that the AppImages in the Releases section do not contain any .pyc files. However, as the instructions in the "Advanced Installation" section probably result in some .pyc files being created, something should be suggested there.

[mxmlnkn]

Here are some quick benchmark comparisons:

wget 'https://github.com/mxmlnkn/ratarmount/releases/download/v1.1.1/ratarmount-1.1.1-slim-x86_64.AppImage'
# Repack it to show that the repacking itself is not the cause of the speedup.
./ratarmount-1.1.1-slim-x86_64.AppImage --appimage-extract
ARCH=x86_64 appimagetool --comp zstd --mksquashfs-opt -Xcompression-level \
    --mksquashfs-opt 22 --mksquashfs-opt -b --mksquashfs-opt 256K --no-appstream \
    squashfs-root ratarmount-repacked.AppImage
time ./ratarmount-repacked.AppImage  # repeated 5x
# real   0m1.056s 0m0.969s 0m0.953s 0m1.020s 0m1.008s
# user   0m0.781s 0m0.729s 0m0.720s 0m0.786s 0m0.764s
# sys    0m0.085s 0m0.086s 0m0.094s 0m0.093s 0m0.102s
stat -c %s ./ratarmount-repacked.AppImage
# 13048312

Compiling all Python files and repacking it:

squashfs-root/usr/bin/python3 -I -m compileall squashfs-root/opt/python3*/lib/
ARCH=x86_64 appimagetool --comp zstd --mksquashfs-opt -Xcompression-level \
    --mksquashfs-opt 22 --mksquashfs-opt -b --mksquashfs-opt 256K --no-appstream \
    squashfs-root ratarmount-compiled-repacked.AppImage
time ./ratarmount-compiled-repacked.AppImage
# real    0m0.365s 0m0.401s 0m0.392s 0m0.342s 0m0.431s
# user    0m0.140s 0m0.173s 0m0.159s 0m0.135s 0m0.177s
# sys     0m0.090s 0m0.082s 0m0.076s 0m0.094s 0m0.098s
stat -c %s ./ratarmount-compiled-repacked.AppImage
# 17615352

Almost a 3 times speedup in startup time! The file size increased from 13.0 MB to 17.6 MB, an increase of 35.0 %.

Repeating this with the full version, which is 6x the size because it bundles all optional dependencies. Many of these dependencies are only imported when actual work is being done, so call it with some arguments.

wget https://github.com/mxmlnkn/ratarmount/releases/download/v1.1.1/ratarmount-1.1.1-x86_64.AppImage
./ratarmount-1.1.1-full-x86_64.AppImage --appimage-extract
ARCH=x86_64 appimagetool --comp zstd --mksquashfs-opt -Xcompression-level \
    --mksquashfs-opt 22 --mksquashfs-opt -b --mksquashfs-opt 256K --no-appstream \
    squashfs-root ratarmount-full-repacked.AppImage
mkdir mounted
time ./ratarmount-full-repacked.AppImage mounted mounted; fusermount -u mounted  # repeated 5x
# real    0m3.273s 0m3.222s 0m3.255s 0m3.392s 0m3.258s
# user    0m2.572s 0m2.579s 0m2.554s 0m2.729s 0m2.596s
# sys     0m0.294s 0m0.259s 0m0.285s 0m0.262s 0m0.271s
stat -c %s ./ratarmount-full-repacked.AppImage
# 63678968
# Compiled and repacked as shown above:
# real    0m1.361s 0m1.107s 0m1.237s 0m1.102s 0m1.314s
# user    0m0.703s 0m0.546s 0m0.580s 0m0.509s 0m0.626s
# sys     0m0.217s 0m0.211s 0m0.224s 0m0.206s 0m0.239s
stat -c %s ./ratarmount-full-compiled-repacked.AppImage
# 76659192

Basically, the same result, i.e., almost a speedup of 3. The file size increased from 63.7 MB to 76.7 MB, an increase of 20.3 %.