Add optional dependency on `pyvips-binary`

[kleisauke]

Allowing users to install pyvips with binary packages via:

$ pip install "pyvips[binary]"

See: https://pypi.org/project/pyvips-binary/.

Marked as draft due to this notes:

• ~~Perhaps this should be in a separate repository?~~ Done: https://github.com/kleisauke/pyvips-binary
• ~Windows binaries are currently using the -static-ffi variant. The plan is to distribute the -static variant when libvips 8.16 is released. https://github.com/libvips/pyvips/blob/ea769b2dab386a959a4e3cb62fc00b81f3051fc5/pyvips-binary/scripts/cibw_before_build.sh#L14-L16 (this will also require a major bump in NetVips, see e.g. commit https://github.com/kleisauke/net-vips/commit/8856756d39c1d0f88a903bce20374d89bc6573e3)~ Done: https://pypi.org/project/pyvips-binary/8.16.0rc1/

[jcupitt]

Nice!

Yes, maybe a separate repo is better.

[kleisauke]

Done!

https://pypi.org/project/pyvips-binary/ https://github.com/kleisauke/pyvips-binary

[kleisauke]

... here are some testing notes:

$ pkg-config --exists --print-errors vips
Package vips was not found in the pkg-config search path.
Perhaps you should add the directory containing `vips.pc'
to the PKG_CONFIG_PATH environment variable
Package 'vips', required by 'virtual:world', not found
$ pip3 install pyvips
$ python -c "import pyvips; print(pyvips.API_mode)"
ModuleNotFoundError: No module named '_libvips'

During handling of the above exception, another exception occurred:

[...]

OSError: cannot load library 'libvips.so.42': libvips.so.42: cannot open shared object file: No such file or directory.

[...]
$ pip3 install pyvips-binary
$ python -c "import pyvips; print(pyvips.API_mode)"
True
$ python -c "import pyvips; pyvips.Image.black(1024, 1024).write_to_file('x.jpg')"
$ file x.jpg
x.jpg: JPEG image data, Exif standard: [TIFF image data, little-endian, direntries=6, orientation=upper-left, xresolution=86, yresolution=94, resolutionunit=2], baseline, precision 8, 1024x1024, components 1

[jcupitt]

Ah fantastic! That's so cool. I'll test here as well.

Shall we put you on the authors list as well? What email addr would you prefer?

[jcupitt]

Is is possible to invert the sense of this option? I'm sure most people should use the netvips binaries rather than the system ones.

So pip install pyvips has the pyvips-binary dependency, and pip install pyvips[system] (something like that?) will use the system libvips.

[kleisauke]

Shall we put you on the authors list as well? What email addr would you prefer?

Feel free to add me to the authors list! I usually use the same email address that's on my GitHub profile. https://github.com/kleisauke/pyvips-binary/blob/deaf1bf1c360e400f8b7b7869d05c4e695c2b4e5/pyproject.toml#L17

Is is possible to invert the sense of this option? I'm sure most people should use the netvips binaries rather than the system ones.

I had the same thought, but it doesn't seem to be possible at the moment. This future possible enhancement is discussed in detail here: https://discuss.python.org/t/4898.

[jcupitt]

Maybe we could just make another pyvips package (pyvips-system?) without a dependency on pyvips-binary?

[kleisauke]

That approach might work, but I'm uncertain if it's the best option for users who prefer relying on system binaries.

For example, Zulip tends to favor using libvips from the package manager (see e.g. commit https://github.com/zulip/zulip/commit/e8f7e281a7d389297783a2185fd0ff09ea4493f5). Additionally, pyvips is commonly used by users who process whole-slide images, something that the pyvips-binary dependency doesn't support (since it uses the -web variant).

The pyvips-binary package also lacks pre-built wheels for less common architectures (such as linux-riscv64, linux-ppc64le, win32-arm64, etc.) and only supports CPython.

Perhaps we should consider catching the OSError mentioned above and re-throwing it with a message suggesting that the optional [binary] package might need to be installed? sharp handles this elegantly, see for example:

Details

node_modules\sharp\lib\sharp.js:114
  throw new Error(help.join('\n'));
        ^

Error: Could not load the "sharp" module using the win32-arm64 runtime
Possible solutions:
- Manually install libvips >= 8.15.3
- Add experimental WebAssembly-based dependencies:
    npm install --cpu=wasm32 sharp
    npm install @img/sharp-wasm32
- Consult the installation documentation:
    See https://sharp.pixelplumbing.com/install
    at Object.<anonymous> (node_modules\sharp\lib\sharp.js:114:9)
    at Module._compile (node:internal/modules/cjs/loader:1460:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1544:10)
    at Module.load (node:internal/modules/cjs/loader:1275:32)
    at Module._load (node:internal/modules/cjs/loader:1091:12)
    at wrapModuleLoad (node:internal/modules/cjs/loader:212:19)
    at Module.require (node:internal/modules/cjs/loader:1297:12)
    at require (node:internal/modules/helpers:123:16)
    at Object.<anonymous> (node_modules\sharp\lib\constructor.js:10:1) 
    at Module._compile (node:internal/modules/cjs/loader:1460:14)

[kleisauke]

This is ready for review now. The last todo note has been fixed with https://pypi.org/project/pyvips-binary/8.16.0rc1/.

[kleisauke]

https://pypi.org/project/pyvips-binary/8.16.0/ :tada:

[jcupitt]

Hooray! This is really great Kleis.

[jcupitt]

Shall we do 3.0.0? I think everything is ready now.

[kleisauke]

Sounds good to me, feel free to release 3.0.0 (coincidentally NetVips 3.0.0 was also released today :tada:).

[jcupitt]

Congratulations!

... I'm being dumb, I don't seem to be able to get pyvips to find its libvips on ubuntu:

$ pip install pyvips-binary==8.16.0
...
Downloading pyvips_binary-8.16.0-cp37-abi3-manylinux_2_28_x86_64.whl (7.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 9.0 MB/s eta 0:00:00
Installing collected packages: pyvips-binary
Successfully installed pyvips-binary-8.16.0
$ 

But it makes a binary with an extra (I guess?) commit number:

$ find ~/testing -name "*libvips*"
./python3.12/site-packages/pyvips_binary.libs/libvips-541602e6.so.42

Then:

$ python3
Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
Traceback (most recent call last):
  File "/home/john/GIT/pyvips/pyvips/__init__.py", line 30, in <module>
    import _libvips
ImportError: libvips.so.42: cannot open shared object file: No such file or directory

[kleisauke]

I wonder if this relates to https://github.com/libvips/libvips/issues/3901#issuecomment-2009154519, which can also be fixed by running pip cache purge. What does ldd $(python -m site --user-site)/_libvips.abi3.so print? I see:

$ pip install pyvips
Defaulting to user installation because normal site-packages is not writeable
Collecting pyvips
  Using cached pyvips-2.2.3-py2.py3-none-any.whl
Requirement already satisfied: cffi>=1.0.0 in /usr/lib64/python3.12/site-packages (from pyvips) (1.16.0)
Requirement already satisfied: pycparser in ./.local/lib/python3.12/site-packages (from cffi>=1.0.0->pyvips) (2.22)
Installing collected packages: pyvips
Successfully installed pyvips-2.2.3
$ pip install pyvips-binary
Defaulting to user installation because normal site-packages is not writeable
Collecting pyvips-binary
  Downloading pyvips_binary-8.16.0-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (2.3 kB)
Requirement already satisfied: cffi>=1.0.0 in /usr/lib64/python3.12/site-packages (from pyvips-binary) (1.16.0)
Requirement already satisfied: pycparser in ./.local/lib/python3.12/site-packages (from cffi>=1.0.0->pyvips-binary) (2.22)
Downloading pyvips_binary-8.16.0-cp37-abi3-manylinux_2_28_x86_64.whl (7.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 8.9 MB/s eta 0:00:00
Installing collected packages: pyvips-binary
Successfully installed pyvips-binary-8.16.0
$ ldd $(python -m site --user-site)/_libvips.abi3.so
	linux-vdso.so.1 (0x00007fd3d253a000)
	libvips-541602e6.so.42 => /home/kleisauke/.local/lib/python3.12/site-packages/pyvips_binary.libs/libvips-541602e6.so.42 (0x00007fd3d1200000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd3d24f4000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fd3d100f000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd3d24e2000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fd3d24dd000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fd3d0c00000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fd3d23f7000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd3d23c9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd3d253c000)
$ python3
Python 3.12.7 (main, Oct  1 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> print(pyvips.API_mode)
True
>>> 

[kleisauke]

The naming of libvips-<unique>.so.42 happens automagically as part of the auditwheel process. Comment https://github.com/lovell/sharp/issues/4095#issuecomment-2394260539 has some background details on this.

[kleisauke]

pyvips-binary 8.16.1 is now available. :tada:

ImportError: libvips.so.42: cannot open shared object file: No such file or directory

Were you able to resolve that? I think it could also fail when you install pyvips after installing pyvips-binary. So, this is fine:

$ pip uninstall -y pyvips pyvips-binary
$ pip cache purge
$ pip install pyvips
$ ldd $(python -m site --user-site)/_libvips.abi3.so | grep libvips
	libvips.so.42 => /lib64/libvips.so.42 (0x00007f9b42400000)
$ pip install pyvips-binary
$ ldd $(python -m site --user-site)/_libvips.abi3.so | grep libvips
	libvips-65e2e1b0.so.42 => /home/kleisauke/.local/lib/python3.13/site-packages/pyvips_binary.libs/libvips-65e2e1b0.so.42 (0x00007ff9f7600000)

However, if you install pyvips-binary first, then pyvips links against a globally installed libvips:

$ pip uninstall -y pyvips pyvips-binary
$ pip cache purge
$ pip install pyvips-binary
$ ldd $(python -m site --user-site)/_libvips.abi3.so | grep libvips
	libvips-65e2e1b0.so.42 => /home/kleisauke/.local/lib/python3.13/site-packages/pyvips_binary.libs/libvips-65e2e1b0.so.42 (0x00007faa32c00000)
$ pip install pyvips
$ ldd $(python -m site --user-site)/_libvips.abi3.so | grep libvips
	libvips.so.42 => /lib64/libvips.so.42 (0x00007fb5c5c00000)

Similarly, installing with pip install pyvips[binary] also links against a globally installed libvips when pkg-config --exists --print-errors vips does not print errors.

$ pip uninstall -y pyvips pyvips-binary
$ pip cache purge
$ pkg-config --exists --print-errors vips
$ echo $?
0
$ pip install pyvips[binary]
$ ldd $(python -m site --user-site)/_libvips.abi3.so | grep libvips
	libvips.so.42 => /lib64/libvips.so.42 (0x00007efe80800000)

This behaviour is probably fine, sharp and NetVips have the same behaviour. However, you can use the SHARP_IGNORE_GLOBAL_LIBVIPS environment variable with sharp to override this.