Add AVIF plugin (decoder + encoder using libavif)

[fdintino]

Resolves #7983

This adds support for AVIF encoding and decoding, including AVIF image sequences.

I've added tests, and integrated libavif into the windows, linux, and mac CI builds. I haven't done anything to integrate with the docker-images repo.

I chose libavif rather than libheif because the former has been embraced by AOMedia and it's what Chromium uses. Packaging support is spotty at the moment, but I expect that to change soon (currently it's in Debian testing, Fedora rawhide, Ubuntu hirsute, and Alpine edge).

A few notes on the implementation here:

• The plugin currently only supports encoding 8-bit images, and all images are decoded to 8-bit RGB(A). I wasn't totally clear on how best to deal with higher bit depths (somewhat related issue: #1888)
• The RGB to YUV conversion isn't exposed to python at all. Chroma subsampling and the presence of an alpha channel make it non-trivial to return decoded images as YCbCr. It would not be difficult to permit encoding from a YCbCr source.
• Since there isn't a way to pass parameters to Image.open (#569), I'm using module globals in AvifImagePlugin.py to make decoder codec choice and chroma upsampling configurable. I suspect there's a better way to do this.

The star.avifs test file is licensed as CC-BY

I linted the C code with the new clang-format settings, but made the following change so that it didn't make PyObject_HEAD and the threading macros look wonky:

diff --git a/.clang-format b/.clang-format
index be32e6d1..300f8e54 100644
--- a/.clang-format
+++ b/.clang-format
@@ -18,3 +18,7 @@ SpaceBeforeParens: ControlStatements
 SpacesInParentheses: false
 TabWidth: 4
 UseTab: Never
+StatementMacros:
+  - PyObject_HEAD
+  - Py_BEGIN_ALLOW_THREADS
+  - Py_END_ALLOW_THREADS

[nulano]

Adding libavif to MSYS2 fails to compile due to a few missing defines (AVIF_CHROMA_UPSAMPLING_AUTOMATIC, AVIF_CHROMA_UPSAMPLING_FASTEST, AVIF_CHROMA_UPSAMPLING_BEST_QUALITY): https://github.com/nulano/Pillow/runs/1683386633?check_suite_focus=true#step:5:77

[fdintino]

@nulano it looks like those defines were only added in libavif 0.8.3. I'll figure some #if version checks around their usage.

[fdintino]

@nulano Is it okay if I cherry-pick your MSYS commit into this PR?

[fdintino]

@radarhere @wiredfool @nulano I think I've addressed all feedback (except for the requests for docs on building), but I've left it up to you all to resolve conversations (or not).

Is this PR generally on the right track? I've held off on writing docs until I've gotten a signal one way or the other.

[fdintino]

Since it's been a month since I asked my question without response, I'll try to reframe it as more specific questions that might be more answerable.

1. Is the general structure of this plugin acceptable? I tried to hew closely to the conventions elsewhere in the repo, so I assume so, but would appreciate confirmation.
2. Is the test coverage sufficient? The gaps are all in the error handling. I'd be happy to try to add test cases for those to make it more complete.
3. I'd appreciate feedback on the encoder settings. For instance: should yuv_format be renamed subsampling to be consistent with the Jpeg plugin? Should I offer a jpeg-like 0-100 quality setting that maps the min/max quantizer? (The colorist library has such a quality setting that still allows qmin/qmax as an advanced override option). Should I eliminate any options? (My vote would be to get rid of qmin_alpha and qmax_alpha).
4. What would you like to see, CI-wise, with this pull request? While waiting for feedback I worked on putting this plugin in its own package, which builds manylinux wheels in the same manner as the pillow-wheels repo. I setup builds for all the codecs supported by libavif, on all platforms, and also added cached dependencies (see pillow-avif-plugin-depends). That includes a crate vendor tarball for rav1e, as I've found crates.io to be too unreliable for frequent CI builds. Would you want to wait until this PR is wrapped up and merged before I opened a pull request against pillow-wheels?
   • Note: I think licensing shouldn't be an issue for anything: AOM, rav1e, dav1d, SVT-AV1 and libyuv are all BSD-2 licensed, libgav1 is Apache, and they all are covered under the Alliance for Open Media Patent License.
   • It might be overkill to include all codecs in the manylinux and windows wheels. In particular, SVT-AV1 is far from ready for prime-time. My recommendation would be to include AOM, dav1d, and rav1e by default. AOM, being the reference implementation, is the most complete and has the highest quality, while dav1d and rav1e are the fastest (setting aside SVT-AV1).

[radarhere]

This might be a libavif bug, but I find that if I run this PR, libavif has stopped working for macOS.

https://github.com/radarhere/Pillow/runs/2201531959#step:8:1174

/Users/runner/work/Pillow/Pillow/depends/libavif-0.8.4/ext/libyuv/include/libyuv/row.h:750:5: error: 'LIBYUV_UNLIMITED_DATA' is not defined, evaluates to 0 [-Werror,-Wundef] #if LIBYUV_UNLIMITED_DATA ^ 1 error generated.

LIBYUV_UNLIMITED_DATA was a change introduced in libyuv in the last month - https://chromium.googlesource.com/libyuv/libyuv/+/ba033a11e3948e4b3%5E%21/#F2

[wiredfool]

We're going to need to add the required libraries to the docker images as well, and we're going to need to add these to the oss-fuzz builder to get fuzzer support.

Might as well make a PR to the Pillow-wheels for whatever needs to happen on build. That will also be potentially helpful for getting the dependencies into oss-fuzz.

[jcea]

I am looking forward to have AVIF support in Pillow. What are the showstoppers to merge this PR?.

Thanks for your work!!

[vanchung1995]

Hello everybody. I'm wondering that how I can custom avif plugin to get return value instead of saving to file? And why in some case, image in avif has larger size (in KB) than origin image like the below image. Origin image has 12.2kb but avif image has 29.7KB?? I used this code:

[radarhere]

As a general answer, if you would like Pillow to place the image contents in a variable rather than saving to file,

from PIL import Image
import io
fp = io.BytesIO()
im = Image.new("RGB", (100, 100))
im.save(fp, 'PNG')
print(fp.getvalue())

[vanchung1995]

As a general answer, if you would like Pillow to place the image contents in a variable rather than saving to file,

from PIL import Image
import io
fp = io.BytesIO()
im = Image.new("RGB", (100, 100))
im.save(fp, 'PNG')
print(fp.getvalue())

Thanks so much. But what about the size of image? I realize that all my 300*200 images in jpeg has double size when convert to avif? How to custom to decrease size of images?

[fdintino]

This isn't really the right forum for these questions. That said: in general AVIF images will compress smaller than JPEG, but that might not always be the case for all images. There are docs included in this pull request listing the settings you can pass to .save(). You might find that decreasing quality will give you the desired file size. You can find those docs here:

https://github.com/python-pillow/Pillow/blob/66660cad793d43e6ed6a8dd7d9dd722065ead676/docs/handbook/image-file-formats.rst#avif

[barboteur]

Hello,

I've just made some tests and effectively, I have to use a quality of 40 in the AVIF format to get a smaller size than WEBP with the default quality used by pillow. And with these quality settings, the image is better in AVIF than in WEBP.

[vanchung1995]

thanks so much all. I tried quality param in save method and I realize same as @barboteur . 40 is a better score

[fdintino]

A quick update on this pull request: I haven't updated or resolved the merge conflicts in the pillow-depends or pillow-wheels repositories because I haven't received any comments here for over a year. Though it's promising to see that it is still occasionally merged against the main branch.

As I mentioned in my previous comment, until this gets merged I will be maintaining this as a separate plugin (pillow-avif-plugin). The CI process there is based off of the one in pillow-wheels. I've made an attempt to keep up with changes there (by, for instance, adding musllinux wheels), but it's not identical so it would need some tweaks before those changes could be incorporated into python-pillow/pillow-wheels#193

[jcea]

I am very interested in getting AVIF support in stock pillow.

[NewUserHa]

.AVIF support lossless. Does this support lossless = True like .webp, and loop = 1?

[fdintino]

@radarhere since you've been pushing to this branch with upstream merges (thanks for that, by the way) I wanted to give you a heads up that I would like to update this PR to have parity with the fixes and changes I've made to pillow-avif-plugin. I think there are other changes in that repository that would address the CI failures in the latest commits.

[vrabaud]

Hi, I am a libavif developer: do you need any help on that PR ?

[fdintino]

@vrabaud that's a very kind offer! I certainly wouldn't turn down some help. I've continued to maintain pillow-avif-plugin but haven't gotten around to pulling my changes there into this branch, and the PR for pillow-wheels basically needs to be rewritten. For the latter task, I have been doing some work to make this more ready for incorporation into Pillow by trying to simplify the CI build. I have had a pull request merged to rav1e to add library artifacts to releases for all of the architectures that Pillow publishes wheels for (and before that, a similar PR to cargo-c). Prior to those being merged they had to be built in virtualization for aarch64 and it took an eternity in GHA. I figured that adding two hours to the wheel build time might have been a deal-breaker for the pillow maintainers.

I was considering doing something similar for libavif: publishing cross-platform artifacts of libavif with all dependencies statically linked. Otherwise updates to the build would need to happen here (most recently for me that meant adding a build and static link for sharpyuv to pillow-avif-plugin). I'd rather not foist that maintenance burden on the pillow maintainers. Would that be a welcome contribution?

[vrabaud]

@fdintino , libavif could use those artefacts for other projects so that would be welcome. Then again, couldn't caching be used here ?

[fdintino]

@vrabaud I have caching for pillow-avif-plugin, but many of my commits are upgrades for libavif or one of its transitive dependencies, which invalidates the caches. With caching there would still be a maintenance burden of compiling and upgrading libavif, rav1e, SVT-AV1, etc. And even if it makes sense to take on that burden for the wheel release workflows, there is still utility in being able to use pre-built libraries for the workflows that run tests.

[vrabaud]

@fdintino , that makes sense. We'll gladly accept a PR for storing artefacts then !

[vrabaud]

Hi @fdintino , anything we can do to help? Thx

[stalkerg]

Can somebody from the core team explain why this very important PR was not reviewed and didn't merge? What we (the community) can do to unlock it? @wiredfool

[aclark4life]

With caching there would still be a maintenance burden of compiling and upgrading libavif, rav1e, SVT-AV1, etc. And even if it makes sense to take on that burden for the wheel release workflows, there is still utility in being able to use pre-built libraries for the workflows that run tests.

@stalkerg Looks like a WIP where the benefit of including in Pillow is still TBD, with specific reference to "maintenance burden". At this point I'd say it's up to @radarhere and @fdintino to find a way forward, if they still believe that forward is the right direction. @wiredfool may have some additional comments. For onlookers, maybe a review of adding AVIF support to Pillow at a high level would help. How strong is the demand? How many are willing to help support it? is the maintenance burden manageable? If the results of the high level review indicate more than just you, @fdintino, @vrabaud (thanks!) and @radarhere are interested in this, that would be good to know.

[jcea]

I am very interested in AVIF support and waiting for it for years. So +1 for me.

[cgohlke]

Besides the maintenance burden, including AVIF would double/triple the size of the Pillow wheels, which might or might not be relevant. I ran into that issue with the imagecodecs package, which bundles libavif.

[mara004]

What is the advantage of supporting AVIF in Pillow in the first place, rather than through an external plugin?

[fdintino]

I don't want to overstate the maintenance burden. I'm a single person and it hasn't been terribly difficult for me to stay on top of libavif updates for pillow-avif-plugin. I'll probably continue to do that for a while—at least as long as I'm working at The Atlantic. We use thumbor, and have it configured to auto-transcode images to AVIF for requests that Accept image/avif. But I certainly wouldn't mind help.

What makes libavif unusual compared to the other pillow dependencies is that it doesn't actually have its own AVIF codec. There are several different AVIF encoding and decoding libraries, that offer different features and have varying performance and compression quality. The value of libavif is the ability to have a single unified API to use AOM, or rav1e, or dav1d, depending on the requirements. I've chosen to ship all supported codecs with the pillow-avif-plugin wheel, but I don't think that pillow would have to do the same. If I had to narrow it down, I would recommend rav1e for encoding and dav1d for decoding. This ought to reduce the impact on the wheel size, and make CI maintenance easier.

I think having AVIF in core pillow might give shape to the implementation of features like high bitrate multichannel images (#1888). There are other features (like supporting YCbCr mode with alpha) that would be nice-to-have, and which would be useful in the implementation of support for next-generation image formats like HEIF and JPEG XL.