Add support for BLAS and LAPACK dependencies (continued)

[mgorny]

This a rebase / continuation of #10921.

So far just gotten it to work again, will look through the previous review next.

[mgorny]

Thanks for your comments. To be honest, I haven't gotten to cleaning up / modernizing the code yet — so far my main focus was getting it to work correctly :-). But pushed the suggested fixes now.

[mgorny]

Okay, I think this is ready for review. FWIU the "linux" test failures are due to the test images not being updated — and the Ubuntu image failure is flakiness of the zig version check.

[mgorny]

Open question: Fedora has OpenBLAS threading variants, and no pkg-config files, i.e. they have 9 OpenBLAS libraries — 3 interface variants (LP64, ILP64 without suffix, ILP64 with suffix) times 3 threading variants (sequential, threaded, OpenMP). Do we want to try to support this? It would imply adding threading: arg to OpenBLAS as well, though i would only be meaningful for Fedora probably.

[dcbaker]

boost already has a threading keyword argument.

edit: Which the interpreter does not allow for...

[mgorny]

Do we agree on going with the current design then? What next steps should I take here?

(the CI workflows are failing because they're running on old images)

[mgorny]

Some more thoughts based on further research I've done in the meantime:

1. I don't exactly like how Netlib uses separate blas and lapack targets, while others use lapack component for it. However, I guess it's kinda inevitable if we want to preserve backwards compatibility with dependency('lapack') — though I suppose we might support both, and possibly deprecate the explicit 'lapack' dep.
2. Sometimes blas and lapack targets evaluate to some other impl (e.g. via symlinks). Should we try hard to actually find Netlib libraries, or actual let blas and lapack evaluate to what the symlinks intended? I guess the latter. Perhaps we could add a dedicated netlib-blas if we really wanted to try to force it (though won't be trivial).
3. At least for OpenBLAS (haven't tested others yet), it is possible for ILP64 lookup to find LP64 library as a last resort. I don't think we can really determine if an arbitrary library with unsuffixed symbols is ILP64 or LP64, but I guess we might do something like requiring a library or a symbol suffix for ILP64, i.e. rejecting plain openblas if it has only unsuffixed symbols as likely LP64.
4. I'm wondering if we should provide some "more automatic" threading: openmp option that would try to determine gomp vs. iomp based on OpenMP implementation used by the compiler. But perhaps it's not worth the effort.
5. As I mentioned above, there are OpenBLAS threading variants on Fedora to consider. If we don't implement that, we will unconditionally use the serial variant (which would be rather suboptimal). However, due to inconsistency in installs, we'd probably need a fallback approach (i.e. "give me OpenMP if available, fall back to whatever openblas is").
6. It would probably make sense to cover FlexiBLAS as well, if only to provide consistent handling of ILP64 interface.

[rgommers]

Thanks for pushing on this @mgorny. Here are my thoughts on your ideas above:

I don't exactly like how Netlib uses separate blas and lapack targets, while others use lapack component for it. However, I guess it's kinda inevitable if we want to preserve backwards compatibility with dependency('lapack') — though I suppose we might support both, and possibly deprecate the explicit 'lapack' dep.

The asymmetry is annoying, but it reflects what's out there in the real world so I can't say it bothers me too much. Also, sometimes plain blas and lapack are not actually the Netlib libraries but stubs/wrappers to compile against (oh, as you also point out in your point 2), so also from that perspective I don't think it's possible to move away from plain dependency('blas')/dependency('lapack'). Deprecating that doesn't seem healthy.

2. Sometimes blas and lapack targets evaluate to some other impl (e.g. via symlinks). Should we try hard to actually find Netlib libraries, or actual let blas and lapack evaluate to what the symlinks intended? I guess the latter.

Agreed, the latter. I don't think it's necessary (nor very easy) to try and distinguish between such wrappers and Netlib libraries.

3. At least for OpenBLAS (haven't tested others yet), it is possible for ILP64 lookup to find LP64 library as a last resort. I don't think we can really determine if an arbitrary library with unsuffixed symbols is ILP64 or LP64, but I guess we might do something like requiring a library or a symbol suffix for ILP64, i.e. rejecting plain openblas if it has only unsuffixed symbols as likely LP64.

There are use cases for both. I think if there'd be a "reject a library with un-suffixed symbol names", it would have to be opt-in for backwards compatibility reasons. As well as for some "build from source and use directly" type use cases; it's probably the more common thing to do for C/Fortran end users, since dealing with symbol prefixes/suffixes can be a fair amount of work. Library authors have incentive to do that work, but scientific end users typically do not.

4. I'm wondering if we should provide some "more automatic" threading: openmp option that would try to determine gomp vs. iomp based on OpenMP implementation used by the compiler. But perhaps it's not worth the effort.

Possibly worth it. It's a fairly minor convenience though, and can always be done later if someone wants that. If we see patterns like if cc.get_id() == 'intel-llvm; threading: 'iomp'; else; threading: 'gomp'; endif in the wild, that's probably a sign that this is needed.

That said, I suspect we'd more need a higher-level "do it all automatically" rather than having other MKL/etc.-specific options.

5. As I mentioned above, there are OpenBLAS threading variants on Fedora to consider. If we don't implement that, we will unconditionally use the serial variant (which would be rather suboptimal). However, due to inconsistency in installs, we'd probably need a fallback approach (i.e. "give me OpenMP if available, fall back to whatever openblas is").

What Fedora is doing is a bit questionable, so I'm not sure how much effort to spend to work around that. What Fedora (well, one stubborn packager, not his colleagues) wants you to do is use FlexiBLAS rather than OpenBLAS. So may as well rely on that rather than more SystemDependency magic. We wouldn't have to think about this if they'd just ship the .pc file like everyone else.

6. It would probably make sense to cover FlexiBLAS as well, if only to provide consistent handling of ILP64 interface.

+1 this seems important. I think the only reason it wasn't there is because I started working on all this before we had FlexiBLAS support in SciPy and before FlexiBLAS itself had ILP64 support with symbol suffixes (IIRC that only landed sometime last year).

[mgorny]

before FlexiBLAS itself had ILP64 support with symbol suffixes (IIRC that only landed sometime last year).

I might have tested wrong but from my short experiments, it still doesn't support ILP64 with symbol suffixes.

[rgommers]

I might have tested wrong but from my short experiments, it still doesn't support ILP64 with symbol suffixes.

You are correct, I remembered wrong. The tracking issue is https://github.com/mpimd-csc/flexiblas/issues/12. It was unblocked last year by https://github.com/Reference-LAPACK/lapack/issues/666 being completed, but the implementation is still TODO.

[lucascolley]

great to see this work! What are the prospects for https://github.com/rgommers/pixi-dev-scipystack/issues/41 in light of this? Are we any closer to figuring out why something seems "off" with the conda-forge distributions, https://github.com/rgommers/pixi-dev-scipystack/pull/23#issuecomment-3051989633 ?

[rgommers]

That's unrelated to Meson (beyond the fact that having the feature in this PR will help a little), so I would keep that discussion at https://github.com/rgommers/pixi-dev-scipystack/issues/41 to not distract the Meson maintainers with it.

[dcbaker]

Sorry, I keep rewriting the dependency code out from under you on this :/

[mgorny]

No problem, so far the changes were easy to follow.

[mgorny]

Rebased.