Pillow icon indicating copy to clipboard operation
Pillow copied to clipboard

Windows build improvements

Open nulano opened this issue 1 year ago • 1 comments

For #6369, close #6554, close #553

In an effort to sync the dependencies available on Linux/macOS (python-pillow/pillow-wheels) and Windows (test-windows.yml) ahead of the next release (#6460), I've compiled a list of dependencies (including indirect ones) below, and included some updates for Windows in this PR.

The full list is: (click to expand)
  • libjpeg via its fork libjpeg-turbo version 2.1.4 is included on all platforms
  • zlib version 1.2.12, included on all platforms
  • libtiff version 4.4.0, included on all platforms
    • uses zlib and libjpeg on all platforms
    • Linux/macOS uses lzma2 version 5.2.6 from xz; added on Windows in this PR and added test
    • no platform uses webp because that is built after libtiff on both platforms; swapped on Windows in this PR and added test
    • no current wheels include deflate, jbig, libjpeg12, lerc, zstd, but cgohlke mentioned at least LERC being included in his wheels in https://github.com/python-pillow/Pillow/issues/5573#issuecomment-872691741
  • webp version 1.2.4, included on all platforms
    • AFAICT its dependencies libjpeg, libpng, libtiff, and giflib are only used for binary conversion utilities, not libwebp. However, on Linux/macOS multibuild ensures these are built before webp. For Windows I moved webp to be built before libtiff.
  • freetype version 2.12.1, included on all platforms
    • uses zlib and libpng on all platforms
    • uses harfbuzz only on Windows because the cyclic dependency is tricky to resolve with GCC
    • only Linux/macOS uses bzip2 version 1.0.8. I tried adding it on Windows, but the driver ftbzip2 appears to have been left out of the Visual Studio project files. It is only used for .pcf.bz2 fonts which aren't that common anyway (especially on Windows).
    • brotli version 1.0.9 added in https://github.com/python-pillow/pillow-wheels/pull/320 for Linux/macOS and this PR for Windows
  • lcms2 version 2.13.1
    • AFAICT its dependencies libjpeg, zlib, libtiff are only used in binary utilities, not the library
  • openjpeg version 2.5.0
    • according to the readme and build files its dependencies zlib, libpng, libtiff, lcms2 are only used in binary utilities, not the library; changed the build flag used on Windows to reflect this (makes builds marginally faster).
  • harfbuzz version 5.1.0 included on all platforms with its dependency FreeType
  • xcb is included on Linux/macOS; not very useful on Windows
  • fribidi is linked at runtime, support shim is enabled on all platforms
  • libimagequant is only used in tests, not included in wheels

Changes proposed in this pull request:

  • Add xz, used by libtiff
  • Build webp before libtiff rather than after so that libtiff can use it
  • Add tests for reading TIFF files with LZMA and WEBP compression, generated with ImageMagick
  • Add brotli for freetype with a WOFF2 font test for #6554 / https://github.com/python-pillow/pillow-wheels/pull/320
  • Append license files to LICENSE before building wheels (on GHA only)
    • I added all included licenses when a readme provided multiple choices, which I later realised was a rejected suggestion for the Linux/macOS wheels. This is both the FTL and GPLv2 license for FreeType, as well as the BSD license mentioned in libjpeg-turbo. Let me know if I should remove these.
  • Restore the py_vcruntime_redist variable. Although I previously thought it was no longer necessary, it turns out this is only true for recent builds of Python (such as the ones on GHA). It is still necessary for older builds such as the one on AppVeyor (I noticed it was emitting a linker warning). It can be safely removed again when Python 3.9.0 is the oldest supported release: https://github.com/python/cpython/blob/v3.9.0/Lib/distutils/_msvccompiler.py#L243

This is enough to get the Windows and Linux/macOS mostly in sync, with bzip2 for freetype, and xcb not enabled on Windows, and webp for libtiff not enabled on Linux/macOS.

Considerations for future PRs:

  • cgohlke's wheels included more libtiff dependencies (e.g. LERC), might be worth adding on all platforms.
  • ~~Add brotli for #6554~~
  • Due to static linking on Windows, combining _webp and _imaging could save about 0.5MB per wheel with the libtiff changes above.
  • Additional testing (e.g. https://github.com/python-pillow/Pillow/issues/5573#issuecomment-877752592); I'm working on a separate PR
  • libpng is failing to detect SSE2 support, should be safe to patch in (even Windows 7 requires SSE2 now)
  • harfbuzz build warning:

    HarfBuzz has a Meson port and tries to migrate all the other build systems to it, please consider using it as we might remove our cmake port soon.

  • openjpeg suggests enabling SSE4.1/AVX2 if available, probably not safe to enable yet

nulano avatar Sep 07 '22 01:09 nulano

no current wheels include deflate, jbig, libjpeg12, lerc, zstd

cgohlke's wheels included more libtiff dependencies

libtiff included in the Windows wheels was built with libjpeg-turbo, libwebp, libdeflate, zstd, lerc, and lzma. JBIG-KIT is GPL licensed.

cgohlke avatar Sep 10 '22 03:09 cgohlke