ENH: move to cibuildwheel

[mattip]

trafficstars

Is your feature request related to a problem? Please describe.

Now that 1.9 is almost out the door, it is time to start thinking about migrating the multibuild Macpython/scipy-wheels build into cibuildwheel here. The first 80% of the should be "copying what numpy does", the last 80% of the work will need to be scipy-specific.

Describe the solution you'd like.

Have cibuildwheel wheel builds directly from this repo

Describe alternatives you've considered.

we could change the Macpython/scipy-wheels repo to use cibuildwheel instead of doing it here, but I think there is value in having this repo directly produce wheels.

Additional context (e.g. screenshots, GIFs)

No response

[charris]

My experience with NumPy is that releases have become easier using cibuildwheel.

[larsoner]

I have also used cibuildwheel on several projects and it's great. I just ported another one over last week, and even the changes to the project in the last 6-12 months made it even smoother.

FWIW, I followed NumPy's pyproject.toml and cibuildwheel infrastructure and it was pretty straightforward! I even needed some extra annoying gymnastics due to my project's complexity and cibuildwheel had hooks that I could use to do it.

So a big +1 from me

EDIT: And thanks NumPy folks for having such clear code that I could easily adapt it!

[tylerjereddy]

I thought I already started in https://github.com/scipy/scipy/pull/15925, just got busy dealing with release and stuff.

[mattip]

Ahh, cool. I searched issues but not PRs.

[rgommers]

This would be nice to start moving forward indeed. My attempts at building 1.9.0 wheels with Meson failed not because of any problems building wheels with Meson, but because multibuild is too painful to deal with.

[tylerjereddy]

Also, I should mention I won't be offended if say someone who currently has a bit of funding takes this over. I spent most of this weekend cramming grant proposals for supercomputer time on Venado (upcoming NVIDIA Grace-Hopper architecture supercomputer).

I suspect I may not get funded until FY23 (October 2022) for OSS contribution work, though I could be wrong, so we may prefer to just have me floating around if it just volunteer time and you really want to drive this forward quickly.

[andyfaff]

I've been experimenting with cibuildwheel locally. As part of the preliminary setup with CIBW_BEFORE_BUILD I installed openblas using the openblas_support.py script as done here.

However, when I run the build step using python3.8 -m build . I get:

Run-time dependency openblas found: NO (tried pkgconfig and cmake) ../../scipy/meson.build:131:0: ERROR: Dependency "openblas" not found, tried pkgconfig and cmake

@rgommers how does cmake find the openblas setup? The include files are in /usr/local/include. The *.so are in /usr/local/lib, and the cmake config files are in /usr/local/lib/cmake/openblas/.

# cat /usr/local/lib/cmake/openblas/OpenBLASConfig.cmake 
SET(OpenBLAS_VERSION "0.3.18")
SET(OpenBLAS_INCLUDE_DIRS /usr/local/include)
SET(OpenBLAS_LIBRARIES /usr/local/lib/libopenblas.so)

EDIT: probably related to https://github.com/scipy/scipy/pull/16750

[mattip]

The best solution would be for the openblas package to provide a proper pkg-config file. Then the build could add it to the PKG_CONFIG_PATH as is done currently for windows https://github.com/scipy/scipy/blob/69f8322c556e8bf2d832c6457c93f8574a1149a6/.github/workflows/windows.yml#L56

This is one of the goals of repackaging openblas in openblas-libs#86.

[rgommers]

CMake detection was indeed fixed in gh-16750

Also, I should mention I won't be offended if say someone who currently has a bit of funding takes this over.

I'll see if we can work this out @tylerjereddy.

[andyfaff]

I'm happy to work on this, I've already put some familiarisation work into this.

[rgommers]

Great, thanks @andyfaff!

[andyfaff]

closed by #16842