Standard Python packaging cannot find native libraries

[thunderbiscuit]

I'm either reporting a bug, or (more likely) looking to better understand how to setup our Python package to integrate the uniffi-rs generated binaries.

At the moment, the standard bindings file as created by uniffi-rs 0.19.3 cannot seem to find our native binaries once packaged.

The loadIndirect() function is what handles this and by default it looks like this:

def loadIndirect():
    if sys.platform == "darwin":
        libname = "lib{}.dylib"
    elif sys.platform.startswith("win"):
        # As of python3.8, ctypes does not seem to search $PATH when loading DLLs.
        # We could use `os.add_dll_directory` to configure the search path, but
        # it doesn't feel right to mess with application-wide settings. Let's
        # assume that the `.dll` is next to the `.py` file and load by full path.
        libname = os.path.join(
            os.path.dirname(__file__),
            "{}.dll",
        )
    else:
        # Anything else must be an ELF platform - Linux, *BSD, Solaris/illumos
        libname = "lib{}.so"

    lib = libname.format("bdkffi")
    path = str(Path(__file__).parent / lib)
    return ctypes.cdll.LoadLibrary(path)

When attempting to use the package, the error we're getting is always something like this:

  File "/Users/user/.pyenv/versions/3.9.10/lib/python3.9/site-packages/bdkpython/__init__.py", line 1, in <module>
    from bdkpython.bdk import *
  File "/Users/user/.pyenv/versions/3.9.10/lib/python3.9/site-packages/bdkpython/bdk.py", line 346, in <module>
    _UniFFILib = loadIndirect()
  File "/Users/user/.pyenv/versions/3.9.10/lib/python3.9/site-packages/bdkpython/bdk.py", line 341, in loadIndirect
    return ctypes.cdll.LoadLibrary(path)
  File "/Users/user/.pyenv/versions/3.9.10/lib/python3.9/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/Users/user/.pyenv/versions/3.9.10/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/user/.pyenv/versions/3.9.10/lib/python3.9/site-packages/bdkpython/libbdkffi.dylib, 0x0006): tried: '/Users/user/.pyenv/versions/3.9.10/lib/python3.9/site-packages/bdkpython/libbdkffi.dylib' (no such file), '/usr/local/lib/libbdkffi.dylib' (no such file), '/usr/lib/libbdkffi.dylib' (no such file)

This has always been the case, and our fix has been to go in and change the loadIndirect() function as an extra, last step when generating the bindings.

Our modified version of the loadIndirect() is the following:

def loadIndirect():
    import glob
    return getattr(ctypes.cdll, glob.glob(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'bdkffi.*'))[0])

At the moment, our final package has the bindings file directly besides the native library:

src/
    bdkpython/
        __init__.py
        bdk.py
        bdkffi.cpython-39-darwin.so

I assume the problem comes from the interplay between where the packaging from setup.py puts the binaries and where the bindings file believes it will be able to fetch them, but I can't seem to make it work, and I'm wondering if you folks have experience releasing Python packages using the uniffi-rs library. We're using a library called setuptools_rust to build our package, and I think our setup.py file is fairly standard otherwise, but I'm looking for ideas on how to fix our packaging script to make it understand where the native binaries should go so that the default loadIndirect() function defined by uniffi-rs can find the binaries everytime.

Links

1. bdk-python
2. bdk-ffi

┆Issue is synchronized with this Jira Task ┆friendlyId: UNIFFI-187

[thunderbiscuit]

Ok so I don't know how I didn't notice this before but the setuptools_rust creates *.so files on my mac M1 and the loadIndirect() function is clearly looking for a .dylib file. I'm not very familiar with the specifics of these file types, but the library works well when deployed on macOS (so the .so file works just as well I guess?).

It might be that we need to ditch the setuptools_rust package (I'm not opposed to that if there is a better solution), but I'd really like to see good examples of how it's done elsewhere before I go and mess with our deployment setup.

[mhammond]

I don't think anyone in the core team has tried to use setuptools with python bindings generated by uniffi, but given a few posts, eg https://stackoverflow.com/questions/32757007/distutils-setup-generate-so-and-not-dylib-on-mac-os-x, it seems like .so might be the default for setuptools and people are working around it in various ways - so this doesn't really sound like a uniffi specific issue. However, I think we'd be open to changes here to make the bindings work better in a setuptools world (but I think we'd also need to better understand what's actually going on to cause this problem in the first place)

[badboy]

Didn't notice this issue until just now. Glean builds for Python, see its setup.py. We don't use setuptools_rust, we have a tiny bit of code that works for us.

[thunderbiscuit]

Thanks for the pointers! It looks like everyone has their own custom workflow/fix, so I wasn't just using the library completely wrong or anything. Our workaround works well at the moment, but I'll try to convince a Python dev to take a look at your Glean setup.py and see if that might be a better solution for us in the long run. Thanks for the help!

[badboy]

Thanks for the pointers! It looks like everyone has their own custom workflow/fix, so I wasn't just using the library completely wrong or anything. Our workaround works well at the moment, but I'll try to convince a Python dev to take a look at your Glean setup.py and see if that might be a better solution for us in the long run. Thanks for the help!

It's probably not :D I'm trying to get rid of our setup.py and switching to maturin. What they do seems to work with UniFFI just fine.