Use only aomenc and dav1d AVIF codecs to decrease wheel size

[fdintino]

trafficstars

Fixes #8856

This removes all libavif codecs except for AOM for encoding and dav1d for decoding. It also builds libavif as a shared library on linux and macOS, which modestly decreases the file size because it allows the linker to omit unused objects from the codec libraries. Building with -Os and -flto reduces the size by another further 0.7-1.5MB.

Below is a comparison of the average CPython wheel sizes by platform for the wheels (MiB) prior to the merge of the avif pull request, the current 11.2.0 wheels, and the wheels from this pull request. An additional few-hundred K could be saved by using aom's decoder instead of dav1d, but I think the benefits of dav1d outweigh those cost savings.

Since Pillow does not yet support multi-channel high bit depth images, I've compiled libaom without high-bit-depth support. This further reduces wheel sizes by ~200KiB.

In absolute terms, the uncompressed libavif shared library file size is somewhere between 2.8MiB (macOS ARM) and 5.0MiB (musllinux x86_64), with the exception of manylinux2014 x86_64 which, because of a gcc bug, is missing some optimizations and so is 7.4MiB.

platform	pre-AVIF	11.2.0	aomenc + dav1d	-flto -Os	further optimizations
macosx_86_64	3.0	10.9	6.9	5.8	5.1
macosx_arm64	2.9	8.5	5.7	4.9	4.5
win_amd64	2.5	13.1	7.4	6.9	6.7
manylinux_2_17_aarch64	4.2	16.3	7.7	6.3	5.6
manylinux_2_17_x86_64	4.3	18.0	8.2	8.2	7.3
manylinux_2_28_aarch64	4.3	16.8	8.0	6.5	5.7
manylinux_2_28_x86_64	4.4	18.7	8.6	7.2	6.3
musllinux_1_2_aarch64	4.3	17.2	8.3	6.6	5.8
musllinux_1_2_x86_64	4.4	18.8	8.7	7.3	6.4

[fdintino]

I'm going to see what effect @wantehchang's suggestion has on the wheel sizes.

[wantehchang]

Frankie: Here are some other ideas to further reduce the binary size.

1. On Linux, use the compiler flags -ffunction-sections -fdata-sections and the linker flag -Wl,--gc-sections. See, for example, https://www.sandordargo.com/blog/2023/07/19/binary-sizes-and-compiler-flags and https://www.sandordargo.com/blog/2023/07/19/binary-sizes-and-compiler-flags.
2. Look at the binary sizes of libsharpyuv and libyuv.
   • If libsharpyuv is big, consider changing -DAVIF_LIBSHARPYUV=LOCAL to -DAVIF_LIBSHARPYUV=OFF.
   • libyuv is critical to performance. If libyuv is big, I will look into how to make it smaller.
3. In libaom, it may be possible to exclude the fallback SIMD optimizations and only include the "highest" SIMD optimizations. For example, if a function has an AVX2 optimization, then exclude its SSE4 and SSE2 optimizations. I will look into this.

[hugovk]

The alternative #8869 removed installation of things like rav1e and svt-av1, are they needed here?

[fdintino]

They have been removed here as well. The only difference between this PR and that one is that this one uses the dav1d decoder instead of AOM's. And it leaves open the possibility of users compiling for more codecs

[wantehchang]

@hugovk Another difference between this PR and the alternative https://github.com/python-pillow/Pillow/pull/8869 is that this PR has additional changes to the build system to reduce the binary size, specifically:

• Change -DBUILD_SHARED_LIBS=OFF to -DBUILD_SHARED_LIBS=ON (except on Windows)
• Change -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=MinSizeRel under some condition (I haven't figured out what the condition is)
• Add -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON

For a fair comparison, these changes should be applied to the alternative https://github.com/python-pillow/Pillow/pull/8869.

[radarhere]

• Change -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=MinSizeRel under some condition (I haven't figured out what the condition is)

It is still using Release for manylinux2014 - see https://github.com/python-pillow/Pillow/pull/8858#discussion_r2029662183

[fdintino]

@radarhere After I merged your change, I noticed that the windows wheels had increased to 7.1 MiB from 6.9 MiB, so I re-added the flag in build_prepare.py.

[fdintino]

It actually looks like macOS is the only platform where CMAKE_INTERPROCEDURAL_OPTIMIZATION=ON results in larger binaries. They are only modestly smaller on some platforms (less than 40KiB on manylinux_2_28 and musllinux), but the LTO binaries are 1.3 MiB smaller on manylinux_2_17 aarch64.

[radarhere]

Another difference between this PR and the alternative #8869 is that this PR has additional changes to the build system to reduce the binary size, specifically:

• Change -DBUILD_SHARED_LIBS=OFF to -DBUILD_SHARED_LIBS=ON (except on Windows)
• Change -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=MinSizeRel under some condition (I haven't figured out what the condition is)
• Add -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON

For a fair comparison, these changes should be applied to the alternative #8869.

To answer this, if I download wheels from the current state of this PR, I get 522.92mb in total. If I remove

-DCONFIG_AV1_DECODER=0 \
-DAVIF_CODEC_AOM_DECODE=OFF \
-DAVIF_CODEC_DAV1D=LOCAL \

I get 488.77mb.

[doublex]

Not in wheels: this is regrettable. AVIF is supported by all modern browsers. Corresponding xkcd: https://xkcd.com/619/

[aclark4life]

Not in wheels: this is regrettable.

@doublex Please see my comments here and clarify if you find the decision "regrettable" even with concerns expressed about wheel size? I'm not sure any feature is worth universally increasing the wheel size by a factor of 10. If we can do it optionally, then I would agree not getting it into this release is regrettable, but I'm not sure how that would work just yet.

[radarhere]

If anyone watching this would like to download prebuilt Pillow wheels with libavif, be aware that you can use the artifacts from the GitHub Actions of this PR, for example by going to https://github.com/python-pillow/Pillow/actions/runs/14424832221 and scrolling down.

[fdintino]

@aclark4life Those comments were made with regard to the initial 11.2.0 release. Those did indeed result in wheels that were as much as 4x larger and uncompressed sizes of 6x or more. Because of various size optimizations, the AVIF feature now only increases both by about 60% (a little more or less, depending on the platform).

I think the door was left open for AVIF ending up in the 11.3.0 wheels, judging from @hugovk's comment here: https://github.com/python-pillow/Pillow/pull/8869#issuecomment-2791718428. The number of people who would be negatively affected by the addition to their virtualenv of a 50 MB file is, I suspect, reasonably large. But 5 MB? For people looking to use Pillow in such resource-constrained environments, we could perhaps provide guidance on how to delete undesired bundled dependencies.

[hugovk]

This PR was about x1.8 in April (https://github.com/python-pillow/Pillow/issues/8856#issuecomment-2827278596), and we're getting close to the next release, how is it looking now?

[doublex]

Yes, please. That would greatly simplify deployment.

[radarhere]

This PR was about x1.8 in April (#8856 (comment)), and we're getting close to the next release, how is it looking now?

The size of Pillow 11.2.1 artefacts (wheels and sdist zips) was 321.03mb

The size from main is currently 374.13mb - it's expected that would go up though, as we now include Python 3.14 beta wheels.

The size from this PR is currently 591.47mb - so that's 1.58 times main.

[radarhere]

I expect libsharpyuv doesn't have a license file here because it comes from libwebp.