[unopy] Python packaging

[cvanaret]

The Python bindings unopy are now merged into main. This PR is dedicated to the packaging of unopy. At the moment, setup.py fetches artifacts of external dependencies, compiles and installs unopy.

cc @jules-vanaret @Opt-Mucca @AlexandreMoulti @FrancoisGallard

Big thanks to @jfowkes for sharing his experience!

[cvanaret]

Hi @jules-vanaret @Opt-Mucca @AlexandreMoulti @FrancoisGallard,

I need to wrap up the Python packaging and register the package so that users can install it via pip install unopy. I wrote a draft for setup.py but apparently it's the old-school way of doing things, so I'd like to use a pyproject.toml instead.

I'm still unsure how to handle the third-party dependencies (AMPL Solver Library, BQPD, HiGHS, MA57, MA27). In my setup.py, I fetch precompiled libraries from their GitHub repos and link them with cmake. That works but it doesn't look safe to have some arbitrary code running under the hood like a black box.

My best guess would be to pass the local paths to the dependencies to pip, have the pyproject.toml transfer them to cmake which then links the local libraries. Something like that (if that's even supported):

pip install unopy -D ASL=local_path -D BQPD=localpath ...

Not super sexy, but it should do the trick. What do you think?

[AntoineD]

Hello,

@FrancoisGallard asked me if I could give advises since I'm used to package python codes.

To make it easier for the end users, you may embed the dependent libs in the python package.

To make it easier for you, you may use cibuildwheel for building and testing the package. You may also be able to use better build backends with less boilerplate by using pyproject.toml instead of setup.py, using meson may also make your life easier than cmake.

[cvanaret]

Hi @AntoineD,

Thanks for your input and my apologies for getting back to you so late.

I'm definitely going to try and write a pyproject.toml for Uno. The issue is that some third-party dependencies are available only as precompiled binaries (BQPD), or must be used with a license (HSL). So it's not something I can easily link in a pyproject.toml (I had to do that manually in my setup.py). My suggestion on Nov 6 was to link these libraries locally, whenever they are present on the user's machine. We could pass their paths to the pip install command, so that the available dependencies are linked to the unopy Python package once it's compiled. I hope that makes sense!

[AntoineD]

Hi @AntoineD,

Thanks for your input and my apologies for getting back to you so late.

I'm definitely going to try and write a pyproject.toml for Uno. The issue is that some third-party dependencies are available only as precompiled binaries (BQPD), or must be used with a license (HSL). So it's not something I can easily link in a pyproject.toml (I had to do that manually in my setup.py). My suggestion on Nov 6 was to link these libraries locally, whenever they are present on the user's machine. We could pass their paths to the pip install command, so that the available dependencies are linked to the unopy Python package once it's compiled. I hope that makes sense!

You can embed those libraries as long as their licenses allow it. For instance, libgfortran is typically embedded in numpy, scipy distributions.

If the license forbids such embedding, you may eventually create a standalone distribution for those, and make you package depend on them. For instance, even though it does not provide a python API (just a cli that wraps a subprocess call to the binary), swig has a python distributions to ease building packages in isolated build environments without having to rely on an external installation of swig.

I doubt that requiring the end user to manage the installation of the library and get it working with a distribution will reliably work and encourage the adoption of this package.

[cvanaret]

@AntoineD you're absolutely right, asking the user to manage dependencies is not the right idea.

The best option at the moment would be to build artifacts in CI with cibuildwheel and publish them to PyPi automatically. We should be able to link static libraries (the precompiled binaries of BQDP are under the MIT license) in the Github action, just before building the wheels. Since the HSL solvers require a license, we'll use dynamic loading to detect the library at runtime.

[cvanaret]

Wheels built with BQPD on Ubuntu and macOS 🥳

[cvanaret]

Wheels built with BQPD and HiGHS on Linux and macOS. The unopy CI test works.

However, it fails with MUMPS_static: the linker claims that it cannot find libblas.a and liblapack.a, even though they're correctly provided (as in unit-tests-linux.yml):

  -- Found BLAS: interfaces/Python/third_party/mumps/lib/libblas.a
  -- Found LAPACK: interfaces/Python/third_party/mumps/lib/liblapack.a
...
  /opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: cannot find -linterfaces/Python/third_party/mumps/lib/liblapack.a: No such file or directory
  /opt/rh/gcc-toolset-14/root/usr/libexec/gcc/x86_64-redhat-linux/14/ld: cannot find -linterfaces/Python/third_party/mumps/lib/libblas.a: No such file or directory