AcademySoftwareFoundation
[HELP] library conflicts with official Python wheels in 3dsMax
@zachlewis This is the follow-up issue based on some initial request on the slack channel by @simonjokuschies.
Hi all, I am running into an issue importing the OpenImageIO python binding in 3DsMax. The python binding gets installed via UV directly from PyPi. Maybe someone can lead me into the right direction how to solve this. 3DsMax-2024 already ships with its own OpenImageIO.dll. When launching 3DsMax-2024 with a custom OpenImageIO (version 3.0.6.1 to be complete here, but the version itself should not matter I think...) and trying to import it I get the following error:
Traceback (most recent call last): File "
", line 1, in File "C:\Program Files\Autodesk\3ds Max 2024\Python\Lib\site-packages\shiboken2\files.dir\shibokensupport_feature_.py", line 148, in _import return original_import(name, *args, **kwargs) File " ", line 8, in find ImportError: DLL load failed while importing OpenImageIO: The specified procedure could not be found. After some troubleshooting, it seems to me that the OpenImageIO.dll that ships with 3dsMax is the issue here, as it gets loaded first. I can confirm that when temporarily removing the OpenImageIO.dll that ships with 3dsMax, besides getting some warnings on launching 3dsMax, I can then actually import my custom OpenImageIO . However, altering the 3dsMax installation shouldn't be the way to move forward here, obviously. So to me, it seems pretty much when I am doing an import OpenImageIO it always loads the OpenImageIO.dll that has been loaded first, which is the one that ships with 3dsMax. I have already tried os.add_dll_directory to point at the custom OpenImageIO installation and also all its sub folders, but that does not seem to do anything. Any thoughts how I could load 3DsMax with a custom OpenImageIO? (edited)
Based on the feedback in slack it feels like there is no easy workaround for us as we do not want to build OIIO ourselves. We also only have partial bootstrapping control over 3dsMax (We can not modify the environment upfront, but only after 3dsMax has been initialized).
The suggestion by @zachlewis was
Hmmm. Maybe we should start using a custom namespace for the official wheels... it certainly couldn't hurt.
Since we don't intend for other libraries to link the OIIO we ship with the Python module, and we don't intend for the Python module to prefer any OIIO other than that with which it ships, I think using a custom namespace for the Python wheels would be safe; although I wonder if other shared libraries that ship with the DCC would still end up causing conflicts... (I can't remember if we build all of OIIO's dependencies statically for the Windows distributions...)
This would be a matter of adding a line to the scikit-build cmake defines in the pyproject.toml, e.g.:
[tool.scikit-build.cmake.define]
OIIO_NAMESPACE = "OpenImageIO_wheels"
- Any suggestions for what namespace we should use, if not "OpenImageIO_wheels"?
Also, for posterity, I'd like to share @JeanChristopheMorinPerso's gist demonstrating how to debug a similar issue re: DCC libraries conflicting with the OIIO wheels' libraries (on Windows).
Also, for posterity, I'd like to share @JeanChristopheMorinPerso's gist demonstrating how to debug a similar issue re: DCC libraries conflicting with the OIIO wheels' libraries (on Windows).
That's great! Thanks for sharing :)
Since we don't intend for other libraries to link the OIIO we ship with the Python module, and we don't intend for the Python module to prefer any OIIO other than that with which it ships,
I feel like that doesn't match our original understanding that "pip install openimageio" could be a convenient way for a wide range of people to get prebuilt oiio installed without having to worry about a full toolchain and all the dependencies being built. I believed that some of those people might not care much (or at least, not exclusively) about the python bindings.
If we're going to back away from that (which we may have good reasons for doing), we must be very clear about documenting that those libraries should not be used for anything but the python bindings, and also we should consider some other route to reliably get single-command binary installs to work for people who want libraries for other reasons (but especially on Windows, where it seems dramatically more burdensome for people to build from source).
It might be that now that we have the auto-build working for a lot of the dependencies, building from source is somewhat easier than it used to be for a lot of people.
I think using a custom namespace for the Python wheels would be safe; although I wonder if other shared libraries that ship with the DCC would still end up causing conflicts... (I can't remember if we build all of OIIO's dependencies statically for the Windows distributions...)
I don't think we force it, but maybe we should? And I'm not sure about what happens on Windows.
* Any suggestions for what namespace we should use, if not "OpenImageIO_wheels"?
OpenImageIO_python ? OpenImageIO_pip ?
Any DCCs embedding OIIO should already be using a custom namespace on their end. So unless they're failing to do that, I'm not sure I understand how it helps to do it on our end.
Any DCCs embedding OIIO should already be using a custom namespace on their end. So unless they're failing to do that, I'm not sure I understand how it helps to do it on our end.
Hmm. That's a very, very good point... @rkoschmitzky, I think the best path forward would be for you to open a ticket with Autodesk, and refer them back to this issue.
I feel like that doesn't match our original understanding that "pip install openimageio" could be a convenient way for a wide range of people to get prebuilt oiio installed without having to worry about a full toolchain and all the dependencies being built. I believed that some of those people might not care much (or at least, not exclusively) about the python bindings.
Ah, yes -- you're correct, of course! I feel like if we do intend to support this, there's stuff we should do to help developers along, like including the dev components, providing metadata to downstream scikit-build-based builds, maybe provide a command line tool for exposing absolute lib / include / bin paths... or we could build a special platform-specific-python-version-agnostic "openimageio_dev" package providing everything but the python bindings, which we could test by using that package to (quckly!) build the regular "openimageio" packages... anyway, a little outside the scope of this thread.
It might be that now that we have the auto-build working for a lot of the dependencies, building from source is somewhat easier than it used to be for a lot of people.
Also a good point. Which makes it stupid easy to build OCIO bindings with a custom namespace using pip or uv with a one-liner, e.g.:
pip install git+https://github.com/AcademySoftwareFoundation/OpenImageIO \
--config-settings=cmake.args="-DOIIO_NAMESPACE=OpenImageIO_foo;-DIGNORE_HOMEBREW_DEPS=ON"
Over at Blender we also do not use the custom ns and I suspect most projects really don't for their dependencies in general? It caused us grief recently with usd as we already expose our version of usd to the user through python, but some folks still wanted to pip install it for, mostly, unnecessary reasons.
I'm not opposed to making our python wheels use a custom namespace, or even to have the python wheels utilize a fully static libOpenImageIO, if that results in a net reduction in headaches for people. If we do that, then we definitely should explain that it's for python only and also definitely not distribute any header files in the python wheels, so nobody can accidentally link against it.
But once we do that, I feel like we've re-opened a gap in the set of options for how somebody can do an OIIO install on a machine without a full developer toolchain and set of dependencies.
To recap, the gap is in trying to help people who want OIIO but:
- Don't have a developer toolchain (the right compilers, all dependencies installed or prepared to build them)
- Don't have access to internet except in limited ways -- maybe they can "pip install" whitelisted packages or can download an installer through a browser, but don't have the general facility to allow our build scripts to spontaneously download dependencies from random GitHub repos.
- Don't have permissions to install new tools or dependencies in the system areas of their machine.
- Or simply want a quick binary install without needing to build absolutely everything from scratch (think: somebody's Windows CI job for a project that uses OIIO as a dependency and they don't want to burn 20 minutes building OIIO and all its dependencies on every run).
Maybe the pip install route is not a good fit for these cases after all. That's fine, but we still have a goal of addressing this set of issues somehow.
I definitely think we should continue pursuing ways to use pip to provide OIIO to developers; but I'm beginning to think we should separate concerns, and provide "official" support for developers via an entirely separate package. It's convenient that we can point folks to the shared libraries used in the current bdists, but I think the population of users looking for a prebuilt Python-version-specific module at runtime is distinct enough from the population of users looking to build stuff against a prebuilt C++ library to warrant entirely separate approaches.
For the immediate future, perhaps we should build the Windows wheels against static libOpenImageIO, which may improve stability on that particular platform in other respects; and we can revisit this decision when we start building wheels against libheif, etc.
The most robust solution is to statically link as much as possible, like we do for linux and macOS.
@zachlewis
Hmm. That's a very, very good point... @rkoschmitzky, I think the best path forward would be for you to open a ticket with Autodesk, and refer them back to this issue.
I’ve filed a ticket with Autodesk as requested.
I see this is a broader discussion and understand all the concerns raised. Thanks, everyone.
To move forward and set clear expectations, would it be realistic to provide a statically linked build—potentially as a separate openimageio-static wheel? If so, are we talking weeks or months to get there?
To move forward and set clear expectations, would it be realistic to provide a statically linked build—potentially as a separate openimageio-static wheel?
I think providing fully statically linked builds in some capacity is a good idea.
If so, are we talking weeks or months to get there?
I guess there's three different answers to this:
-
Immediately: there are [very impressive!] unofficial static wheels you can experiment with immediately:
pip install oiio-static-python -
Short term (weeks): if there are no objections, I'd like to set
OIIO_STATIC=1for the Windows wheels builds for the next patch release (or hotfix shortly after), and then switch it back immediately afterward. I suspect -
Long term (months):
- We need to consider the implications of statically linking certain LGPL-licensed dependencies required by certain OIIO plugins (HEIF, AVIF, Raw)...
In an ideal world, the Python bindings would certainly use a fully static OIIO (and its dependencies) and expose no symbols but the minimum that Python needs to make a successful module.
But let's just note that in the practical world, there are two issues to grapple with:
-
Some OIIO dependencies have LGPL components that are considered permissible to dynamically link but not statically link, without our also needing to distribute the combination under LGPL, which in turn means that people wouldn't be able to distribute downstream software that uses our module unless it, too, is LGPL. (To be clear: it is our stated project goal that it should be 100% ok to make and sell commercial software that uses OIIO internally.)
I think, actually, that we already exclude those libraries from the Python wheels. But we did have a goal of figuring out if there was a safe way to do it that would comply with all the licenses. Switching pre-emptively to a fully static setup might be fine right now but would be something we'll need to revisit later if we want to expand to allow those dependencies.
-
Making the PyPI distributed OpenImageIO package use static linkage means that it's only usable as a Python module, and that "pip install openimageio" would not be at all helpful as a 1-step route to get working OIIO libraries that you could use for any other purposes. Maybe that's fine! But if so, we should perhaps come up with an alternate plan for getting people pre-built binaries, since building from scratch (even with auto-build) is a PITA that many users who only want to consume OIIO would prefer not to be burdened with.
I don't have a strong feeling about what to do next, I'll go with your recommendation. I'm just being super sure we remember why we weren't static from the start.
To clarify, I think the OpenImageIO package should continue dynamically linking libOpenImageIO built against static dependencies wherever we can, and linking + bundling shared dependencies where we can't (e.g., to give folks a way to swap in a custom abi-compatible libheif.so, per LGPL), and we should avoid going the fully static route if we can possibly avoid it.
That being said, I'd like to do a statically built release for Windows in the near term because I suspect the wheels will be more stable in other respects as well, and it would be nice to have a version we can refer folks to for troubleshooting problems experienced with past (and future) dynamically linked Windows builds -- and, as you say, we're not currently building against any LGPL-licensed libraries at the moment, so it should be kosher to release a fully static build. But apart from this one release, I think we should continue building (and improving support for) dynamically linked Windows wheels, at least until we run into a strong enough reason not to.
If we do end up deciding to switch to fully static builds for whatever reason, we should definitely provide a separate openimageio_dev package explicitly providing headers / cmake configs / prebuilt libraries (and maybe even cli tools) without providing the actual OpenImageIO Python module.
I'm just being super sure we remember why we weren't static from the start.
- To reduce the file size of each bdist; although at the time, we were building all the CLI tools except
iv. - To reduce the complexity of dealing with the LGPL-licensed dependencies (it's just a coincidence that we haven't gotten around to providing support for libheif / libde265 / libraw self-builds...)
- To use pip as a means for distributing a prebuilt libOpenImageIO for other purposes.
If we can continue doing all three at the same time, I'm all for it. I'm just saying, we could separate concerns into distinct openimageio_tools, openimageio_dev, and openimageio packages, if that ends up being easier to support.
- To use pip as a means for distributing a prebuilt libOpenImageIO for other purposes.
I don't think that's a wise thing to do. We should try to avoid shipping things like this on PyPI. If people want pre-built binaries, they can be attached to the releases as a separate artifact.
I'm just being super sure we remember why we weren't static from the start.
We are already statically linking on macOS and Linux, see: https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/b8fe2612f3b87b71dd462b2f852d1d83a8340c2e/pyproject.toml#L112 https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/b8fe2612f3b87b71dd462b2f852d1d83a8340c2e/pyproject.toml#L119
$ ldd OpenImageIO/OpenImageIO.cpython-39-x86_64-linux-gnu.so
libOpenImageIO.so.3.0.6 => /asd/OpenImageIO/lib/libOpenImageIO.so.3.0.6 (0x00007ffffe539000)
libOpenImageIO_Util.so.3.0.6 => /asd/OpenImageIO/lib/libOpenImageIO_Util.so.3.0.6 (0x00007ffffe3d4000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ffffe0c7000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffffddc4000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffffdbae000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffffd992000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffffd5c3000)
/lib64/ld-linux-x86-64.so.2 (0x00007fffffddc000)
libz.so.1 => /lib64/libz.so.1 (0x00007ffffd3ad000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffffd1a9000)
This shows that we are affectively statically linknig all the dependencies into the shared libraries. We don't statically link libOpenImageIO.so into OpenImageIO/OpenImageIO.cpython-39-x86_64-linux-gnu.so to save space.
As for the LGPL dependencies, we indeed have to be careful to not statically link them. We could leave them as shared libraries, but it'll come at the cose of potentially "broken" OIIO like in this current issue. We could in theory workaround most problems by doing something like this in the main __init__.py file:
import os
import ctypes
# Force loading our own shared libraries first
ctypes.cdll.LoadLibrary(os.path.join(os.path.dirname(__file__), "bin", "mylib.dll"))
import OpenImageIO
but that won't necessarily work in all cases. It'll potentially break (like here) whenever something else imported a module that happens to load a shared library that is not ABI compatible or doesn't have the symbols we need.
Note that the next monthly patch release is scheduled for Tuesday (1 July), so the window is rapidly closing to introduce any changes we want to ship this time around.
and that "pip install openimageio" would not be at all helpful as a 1-step route to get working OIIO libraries that you could use for any other purposes.
To chime into this discussion somewhat unrelated. I was actually not aware that this was possible/intended and I might actually end up switching up some of my projects to do this rather than having to build OpenImageIO from scratch which takes a non-trivial amount of time.
Could you confirm that this is indeed intended and that it's a path I can take going forward as that would simplify setup on my end by a lot!
@EmilDohne I think the summary of where we are is like this:
- We did originally envision "pip install", in addition to the main way Python developers would get OIIO, as being a convenient way for anybody to get a binary install including all dependencies from a single command that just works on almost all platforms.
- In practice, it had some shortcomings: it didn't support all the formats (licensing issues to be worked out), it wasn't as fully optimized (I think we were compiling for space rather than performance), and it didn't contain all parts of OIIO (again, a casualty of trying to keep the wheels small).
- Also, per this ticket, it seems that there are some edge cases where maybe we don't want the Python wheels to have symbols or library names that clash with the regular OIIO that other apps use?
- So now, we're thinking that maybe we should think of the wheel as for python users only, but that means that we should probably look for another way to automatically produce pre-built binaries with each release.
Thanks for the reply! In that case I'll keep following the developments here and on other issues and keep going with my vcpkg setup for building openimageio :)
@zachlewis
I guess there's three different answers to this:
Immediately: there are [very impressive!] unofficial static wheels you can experiment with immediately: pip install oiio-static-python
Yes, we've tested these already and didn't encounter any issues. :) Luckily we can postpone a decision on our end for a bit on how to deal with this, but we are glad that this alternative exists.
That said, we’d still prefer to stick with the official openimageio package to avoid any potential dependency conflicts between it and oiio-static-python in our package ecosystem.
As a side note, I really appreciate that the wheels include oiiotool, which nowadays makes it super easy to run commands like:
uv tool run --from openimageio oiiotool
After working a bit on my own project I may have stumbled upon a solution that seems to work quite nicely.
Cibuildhweel doesnt by default repair the wheels for windows (copying the dlls to the wheel directory). From what I gather openimageio has its own implementation/workaround for this. Recently I found the delvewheel package which not only tackles this step, but it also uniquely hashes the dlls and patches the pyd to link to those instead. This would avoid any other OpenImageIO.dll conflicting if it has a different version.
Obviously 3dsmax should also pre/post-fix their dlls but I think this could be a good proactive step to allow other packages to fail to do this without causing these issues
EDIT:
The delvewheel package is also officially recommended by cibuildwheel for this step but not yet the default