Support cross compiling

[isuruf]

Even though meson supports cross compiling, it seems meson-python does not.

https://github.com/mesonbuild/meson-python/blob/78861f5ee4e5257cdebe3ab9faf84027466f1c0b/mesonpy/init.py#L313-L348 assumes that the extension is run on the same interpreter. We use crossenv to cross compile in conda-forge and crossenv monkey-patches some things but monkey-patching importlib.machinery.EXTENSION_SUFFIXES does not seem like a good idea.

cc @h-vetinari, @rgommers, @eli-schwartz

[rgommers]

Thanks for identifying that issue @isuruf. It looks like we need to avoid this extension check, and also add a basic cross-compile CI job. I was just looking at that for SciPy. I'm not that familiar with crossenv, but I think that for meson-python CI we need a cross-compilation toolchain (maybe from dockcross) and also both host and build Python already installed - crossenv doesn't do that for you if I understood correctly. So we need a Docker image with those things - perhaps from conda-forge?

[dnicolodi]

I don't think calling this a bug is fair. Cross compilation is not supported by the Python packaging ecosystem. And apparently "crossenv monkey patches some things" to make it work, whatever this means, given that PEP 517 build are usually done out-of-process. The detection of the stable ABI is just one of the places where we introspect the running environment, and Meson does too. Python does not offer a supported way to obtain the information required to build an extension module for another interpreter, let alone for an interpreter running on another platform. If you need support for cross compilation, please bring the issue to the attention of the CPython developers.

[rgommers]

Fair enough, Python's support is very poor. But right now moving from setuptools to meson-python is a regression, because crossenv does this ad-hoc patching of just the things that setuptools needs. So it is important and we need to fix it. Ideally both with and without crossenv.

We're not that far off, I'm just messing around with trying to get a Linux x86-64 to aarch64 cross compile to work for SciPy with:

dockcross-linux-arm64 bash -c "python -m pip install . --config-settings=setup-args=--cross-file=/work/cross_aarch64.ini"

I suspect it only requires a few tweaks.

[dnicolodi]

The first obvious issue I can think about is that we use sys.maxsize to determine if the target Python interpreter is 32 or 64 bits. How do you plan to make that work? AFAIK setuptools uses the same check, thus setuptools does not support cross compiling between 32 and 64 bit architectures.

[eli-schwartz]

The detection of the stable ABI is just one of the places where we introspect the running environment, and Meson does too.

The difference is that meson specifically executes an external python installation with a json dumper script, in order to scrape for information which meson knows it may need. It looks like meson-python checks for this information in-process instead, which means that it is bound to the same version of python that it is running inside.

Cross compilation has two challenges:

• compile for something other than what you are
• compile for something you cannot exec on your hardware/software stack

[dnicolodi]

Well, meson-python implements PEP 517, which does not have any provision for cross-compiling. It assumes that the Python interpreter used for the build, is the one you are building for. More generally, there is no way to generate wheel tags for an interpreter that you cannot execute. And if you can execute the Python you are building for, why not use it to run meson-python? I know all this is not ideal. But supporting these interfaces with these constraints is what meson-python is set up to do. If we want to build a tool for cross compiling Python wheels, it would have to look very different.

[eli-schwartz]

Sure, I do understand and empathize with those issues. There is finally some interest in getting this to work once and for all, though, I think. :)

And if you can execute the Python you are building for, why not use it to run meson-python?

FWIW this is actually a complex topic. Particularly worthwhile to note:

• qemu makes it possible to more or less fully generically emulate other machines, with the restriction that this is Linux-specific
• WINE on Linux can emulate Windows

But actually doing so is slow. So you actually don't want to do this, at least not more than you can get away with. So, there's a benefit to emulating just the json dumper and then running the rest of the build natively.

[dnicolodi]

So, there's a benefit to emulating just the json dumper and then running the rest of the build natively.

I completely agree, but while it makes sense for Meson to work this way, I think it would be overkill for meson-python. But, because PEP 517, we don't even have to think about it: the interfaces we need to implement do not support this.

[rgommers]

There's several levels of making things work here:

1. Make it work with crossenv, with the same level of support as setuptools has
2. Make it work by figuring out what we actually need to know (wheel tags, `sys.maxsize & co) and then allowing a user to put that into a cross file
3. Make it work out of the box without the user having to do (2) (this one requires stdlib support)

(1) and (2) should be feasible in a shorter time frame. (3) is going to be a lot more painful.

[h-vetinari]

If you need support for cross compilation, please bring the issue to the attention of the CPython developers.

By necessity, conda-forge has built a lot of its packaging around cross-compilation (i.e. there aren't native osx-arm64 CI agents, so we need to cross-compile from osx-64). These builds might even flow back to the non-conda world occasionally.

So it's a classic case of Hyrum's law, where a lot of things have grown up around an implicit interface (in this case the possibility to monkey-patch in cross-compilation), that we cannot switch the conda-forge builds for scipy to meson unless we solve the cross-compilation part somehow.

I don't mind carrying patches or workarounds for a while (i.e. it doesn't need to have a sparkling UX), but it would be very beneficial to have it be possible at all.

[dnicolodi]

Cross compiling for an arm64 target on a x86 build machine on macOS is already supported with the same interface used by setuptools.

For other user cases, I'm afraid that making the monkey-patches required to make cross compilation work with setuptools also work for meson-python is not possible, unless someone defines somewhere the interfaces that we can use and the ones we cannot. Even then, the thing would be extremely fragile without a clear definition of how the interface that we can use work when cross-compiling.

importlib.machinery.EXTENSION_SUFFIXES here is only as a safety check. We can remove it. But I would be very surprised if there are no other things that break.

[rgommers]

Even then, the thing would be extremely fragile without a clear definition of how the interface that we can use work when cross-compiling.

This is true, but the good thing is that the build envs are much better under control, and regressions are not as painful. We're looking to support distros here, not make pip install mypkg --cross from source work for end users.

[eli-schwartz]

It saddens me a bit that people seem to think crossenv is the definition of how to cross compile. There are people who have been cross compiling python modules without knowing that crossenv exists (or I think in one case, being vehemently convinced that crossenv is horrible and the worst thing possible for the cross compilation community 🤷‍♂️).

I think the reality is some combination of "a number of different groups have independently discovered some key aspects, and have different ideas how to do the rest".

• There's no such thing as a crossenv interface
• nor a cross-compile setuptools interface regardless of framework used to run setuptools

Meson has a cross-compile interface. Meson defines how to cross compile a meson project.

Frameworks used for cross compiling, including but not limited to crossenv, yocto, buildroot, voidlinux, etc, are responsible for interacting with the Meson cross-compile interface, and that is all. Meson, in turn, considers its cross compile interface to be "run a python interpreter and dump a specific list of values" -- this isn't well documented in the manual, but it does exist.

(Those projects may have also homebrewed their own cross compile interface for setuptools, but that really doesn't matter for either meson-python or for meson. At the end of the day, the tricks they use are irrelevant to meson except for the sysconfig tweaking, and that's just parallel evolution, not something owned by a specific tool.)

IMHO meson-python shouldn't fail to package a project that meson has successfully cross compiled, and for that reason I'm happy to see the check being removed. :)

If you want to validate that the ext suffix matches the binary architecture, that seems like a job for auditwheel or something.

If you need to generate a wheel tag, I actually feel a bit like that doesn't much matter. If you're building for local use you just immediately extract the results and discard the wheel tag in the process (and I assume conda has the same basic rationale to not-care) and again, I thought this was what auditwheel is for. If it can fix the manylinux version, I'd assume it can also fix a broken CPU architecture... can we just fall back on a nonsense wheel tag like "unknown"? Or lie and call it "none-any"?

[rgommers]

It saddens me a bit that people seem to think crossenv is the definition of how to cross compile.

I don't think that. crossenv is indeed just one of the ways, and seems to be a pragmatic hack to work around Python lack of support.

If it can fix the manylinux version, I'd assume it can also fix a broken CPU architecture... can we just fall back on a nonsense wheel tag like "unknown"? Or lie and call it "none-any"?

I agree with most of what you wrote, but not this. auditwheel is specific to manylinux-compliant wheels, and manylinux is not appropriate or needed in a number of circumstances. We do need to generate a correct wheel tag. It shouldn't be that hard.

It seems to me like meson-python does need to know that we're cross compiling. Detecting whether --cross-file is present in config_settings is probably reliable (covers everything except for the macOS environment variable way)? If so, could we just read the host machine section of the cross file, and generate the wheel tag from that?

[dnicolodi]

@eli-schwartz I agree with you on all points, except one: the tools that takes a Meson build directory and packs it up in a wheel is not meson-python, but something else, that may or may not share some code and be implemented in the same project as the current meson-python.

meson-python define itself has an implementation of PEP 517 https://peps.python.org/pep-0517/ In the interfaces defined in PEP 517 there is nothing that allows cross-compilation: it implicitly assumes that the wheels are being built for the Python interpreter running the build process. This is one of the reasons why solutions for cross compiling wheels have the taste of hacks: they need to work-around this interface limitation. AFAICT, auditwheel has the same core assumption, thus I don't think it can fix wheels built for an architecture that is not the one where it is being run.

Building on @eli-schwartz consideration that the correct cross-compilation interface is the one provided by Meson, we need a tools that allows access to that interface (forgetting PEP 517). However, wrapping Meson in another tool dedicated to build Python wheels is not necessary, what the tool needs is just the Meson build directory. I'm thinking to something that could be run like this:

meson setup $build_dir --cross-file $crossbuild_definition
meson compile -C $build_dir
meson-wheel-it-up $build_dir --tag $wheel_tag

Where meson-wheel-it-up is just an implementation of meson install that packs the build artifacts into the right format.

[dnicolodi]

Detecting whether --cross-file is present in config_settings is probably reliable (covers everything except for the macOS environment variable way)? If so, could we just read the host machine section of the cross file, and generate the wheel tag from that?

This would require determining which architecture we are building for from the compiler executable paths. I'm sure it can be done, but the user knows for which architecture they are building, they can just tell the wheel packaging tool about it. Also, it would require determining which flavor of python interpreter (cpython/pypy/pyston,... and the relative version) we are building for from the executable path. Also this seems an information that the user is in a better position to tell us.

More problematic is the handling of build dependencies and (optional) build isolation implemented by PEP 517 frontends. You most likely do not want that for cross compiling. Build dependencies for cross compilation need to be correctly handled considering which dependencies are to be run on the host (cython, pythran) and which are libraries for the target (numpy, scipy, ...). PEP 517 frontends cannot do that, and they mostly get in the way.

[rgommers]

the tools that takes a Meson build directory and packs it up in a wheel is not meson-python, but something else, that may or may not share some code and be implemented in the same project as the current meson-python.

This is a little confusing to me. I'm not sure what you have in mind here exactly. Whatever other project it is, I think that is an implementation detail under meson-python. From my perspective only have two things: Meson as the build system, and meson-python as the layer between pip & co to build sdists and wheels. And the goal of meson-python is to make use of Meson by Python projects as seamless as possible.

What is and isn't in PEP 517 isn't that relevant, there's --config-settings as an explicit escape hatch for anything else that's not directly supported by a standard. And that includes cross-compiling. In fact, proper support for cross-compiling is one of the major benefits of moving from distutils & co to Meson. We certainly get a lot of bug reports and questions about it for NumPy and SciPy, and in the past (pre Meson) I've always answered "don't know, distutils doesn't support that, good luck and if you learn something please share". I now want proper support for it.

AFAICT, auditwheel has the same core assumption, thus I don't think it can fix wheels built for an architecture that is not the one where it is being run.

auditwheel was brought up by both of you, but it really isn't relevant. Its job is specifically to deal with distributing wheels on PyPI. There are many packaging systems other than PyPI, so we cannot rely on auditwheel for anything.

that the correct cross-compilation interface is the one provided by Meson, we need a tools that allows access to that interface

It's not really an interface in the sense that meson-python can use it - but I don't think there's a need for that. We only need a few fragments of information. Basically metadata for platform, OS, and interpreter flavor and ABI - I'm not sure that there's much beyond that. So we should just figure that out from the info config_settings incoming data.

This would require determining which architecture we are building for from the compiler executable paths.

Not really - there's a section like this in the cross file:

[host_machine]
system = 'windows'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

it would require determining which flavor of python interpreter (cpython/pypy/pyston,... and the relative version) we are building for from the executable path. Also this seems an information that the user is in a better position to tell us.

That's a good point. I think the requirement here is "build interpreter kind == host interpreter kind" for now (in practice, the main demand is CPython). Possibly there's a need to add the host interpreter to the cross file - let's cross that bridge whne we get to it.

More problematic is the handling of build dependencies and (optional) build isolation implemented by PEP 517 frontends.

No one is going to use build isolation with cross compilation I think. It'll be something like:

python -m build --wheel --no-isolation --skip-dependency-check -C--cross-file=cross_aarch64_linux.ini

There's nothing for us to do here to deal with build isolation AFAIK.

[dnicolodi]

I think the requirement here is "build interpreter == host interpreter" for now

The problem raised in this issue is about the host and the build interpreter being different (if they are the same, of course importlib.machinery.EXTENSION_SUFFIXES needs to be valid for the extension modules being packaged). If the host interpreter is the same as the build interpreter, meson-python already works just fine, AFAIK.

No one is going to use build isolation with cross compilation I think. It'll be something like:

python -m build --wheel --no-isolation --skip-dependency-check -C--cross-file=cross_aarch64_linux.ini

I don't see what going through build and the PEP 517 interfaces gives you in this case. If you need to pass tool-specific command line arguments (the -C--cross-file, which by the way needs to be -Csetup-args=--cross-file=cross_aarch64_linux.ini) you don't even have the advantage of having a tool-agnostic command line interface. Furthermore, you make optional arguments (--no-isolation and --skip-dependency-check) mandatory. It still looks like an hack and a magic incantation more than a solution.

[rgommers]

I think the requirement here is "build interpreter == host interpreter" for now

The problem raised in this issue is about the host and the build interpreter being different

I meant "the same kind, so both CPython or both PyPy". That's a reasonable default, and I think conda-forge's cross compiling jobs for PyPy do that.

I don't see what going through build and the PEP 517 interfaces gives you in this case.

It's the only way to get the .dist-info metadata and a wheel format output that you need. I keep on seeing this confusion, but --no-build-isolation is not niche, it's extremely important (most non-PyPI packagers need this, and I certainly use it more often than not also for local development) and we should treat it on par with the default.

Furthermore, you make optional arguments (--no-isolation and --skip-dependency-check) mandatory. It still looks like an hack and a magic incantation more than a solution.

They're already mandatory for many use cases. --no-isolation was a choice for a default that optimized for "build me a wheel to distribute on PyPI". Many/most other use cases require no isolation. It is definitely not a hack.

[dnicolodi]

It's the only way to get the .dist-info metadata and a wheel format output that you need.

The PEP 517 backend is responsible for generating the .dist-info https://github.com/mesonbuild/meson-python/blob/f9dc18f85475e80e4cba31105cbe6a4e42660b78/mesonpy/init.py#L556-L568 It has nothing to do with using a PEP 517 fronend to invoke the wheel packaging tool.

[rgommers]

@dnicolodi I'm not sure what you mean there. meson-python is that backend, and the only way to use meson-python is via a frontend like pip or build.

to invoke the wheel packaging tool.

It seems like you have a conceptual model here that I do not understand. If I understand you correctly, you have something other than Meson and meson-python in mind, but I don't know what that would be.

[eli-schwartz]

From my perspective only have two things: Meson as the build system, and meson-python as the layer between pip & co to build sdists and wheels.

My perspective is a bit different, but I think that's because I approach the whole issue from a different direction

I view meson-python as two things:

• a producer of dist-info metadata, and python distribution (the keyword for "importable thing with metadata) constructor
• binary archiver for pip-compatible installer bundles

And you mention "pip & co" but I think it's a bit simpler, and reduces down to "pip". Or maybe "PyPI".

As I alluded to above, for local builds a wheel is a waste of time, and so are platform tags. What you actually want is a site-packages, and wheels are just a way for build backends to tell installers what files to install. System package managers like conda, dpkg, rpm, pacman, portage, etc. don't care about wheels, because they have better formats that provide crucial cross-domain dependency management among other things. They also, consequently, have better ways to define platform tags than, well, using anything as ill-defined and non-granular as platform tags.

And what even looks at platform tags anyway? Just pip, basically... and, importantly, only in the context of downloading from PyPI.

...

From the perspective of another package manager trying to repackage software produced by the pip package manager, wheels look like those makefiles where "make" creates a tar.gz file that you have to untar, and don't provide a "make install" rule.

auditwheel was brought up by both of you, but it really isn't relevant. Its job is specifically to deal with distributing wheels on PyPI. There are many packaging systems other than PyPI, so we cannot rely on auditwheel for anything.

But this does in fact tie into my belief that platform tags are also specifically to deal with distributing wheels on PyPI.

The rest of the time it is a vestigial organ. While it can't hurt to get it correct where possible, this shouldn't come at the sacrifice of important functionality like generating a successful build+install. When in doubt, apply either a validating placeholder or a genetic tag that is technically correct but provides no information, like "linux" (is that a valid tag? Do you need the CPU architecture?)

The result doesn't matter, if you're locally building the wheel, locally installing it, and locally creating a conda package out of it.

I don't see what going through build and the PEP 517 interfaces gives you in this case. [...] you don't even have the advantage of having a tool-agnostic command line interface.

Because setuptools install had the implementation behavior of a) executing easy_install instead of pip, b) producing egg-info instead of dist-info, and made the unusual development decision to claim that they can't change this because projects might be depending on egg-info specifically, therefore "we will make you stop using egg-info by using bdist_wheel, dropping install, and breaking your project anyway". They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

Because of the privileged position setuptools held in the ecosystem, this has become the new model for build backends, namely, that they shouldn't provide a command line. And ex post facto, this has been reinvented, rather than being due to peculiarities of egg-info, to instead be due to a host of imagined reasons for why command lines are bad, even as an alternative. ;)

The result is that the advantage you get from going via build and a series of command-line arguments is "it's a program that can generate a library API call to a build backend library".

Flit is fighting this trend, as flit_core.wheel provides a command line. However I think the main motivation there is to make it easily bootstrappable (you don't need to build build without having build, before you can build flit_core), not about how noisy the command line is.

It's a weird quirk but ultimately not a huge deal now that the core infrastructure has to some degree settled on flit.

Note that any program which uses PEP 517 to generate a library API call to build a wheel, is a PEP 517 frontend. But not all PEP 517 frontends support build isolation, or default to it. For example, Gentoo Linux has written an internal frontend called gpep517, which is a "Gentoo pep517", that relies on the knowledge that Gentoo never ever wants build isolation.

Yes, we're now seeing a proliferation of incompatible frontend command lines as a retaliatory response to the unification of backend APIs. And no, frontends aren't trivial to write either.

[rgommers]

As I alluded to above, for local builds a wheel is a waste of time, and so are platform tags

Yes and no. All you do with the final zipfile is unpack it and throw it away, so from that perspective it doesn't do much. However, you do need the metadata installed. So a meson install into site-packages gets you a working package, but you're missing the metadata needed to uninstall again. A wheel in this respect is like a filter that ensures everything one needs is present, so uninstalling/upgrading with pip afterwards works, as do things like using importlib.resources.

System package managers like conda, dpkg, rpm, pacman, portage, etc. don't care about wheels

Not as a distribution format, but they do in practice. These tools very often run pip install . --no-build-isolation to build the package. And then as a final step repackage it into a .conda, .rpm or whatever their native format is.

They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

I completely agree that that's a mistake, and I appreciate Meson's nice CLI. But I think that's a mostly unrelated point here. If meson had a command that'd yield the exact same result as pip install . --no-build-isolation then I'd say we could use that and there'd be no need to go through a wheel. But there's no such command (yet, at least).

[eli-schwartz]

Yes and no. All you do with the final zipfile is unpack it and throw it away, so from that perspective it doesn't do much. However, you do need the metadata installed. So a meson install into site-packages gets you a working package, but you're missing the metadata needed to uninstall again. A wheel in this respect is like a filter that ensures everything one needs is present, so uninstalling/upgrading with pip afterwards works, as do things like using importlib.resources.

Right, like I said, meson-python does two things, and one of them is producing that metadata, and the other one is producing that wheel.

I completely agree that that's a mistake, and I appreciate Meson's nice CLI. But I think that's a mostly unrelated point here.

Right, this is very much a side topic in response to @dnicolodi's question about "I don't see what going through build and the PEP 517 interfaces gives you in this case" and magic incantations.

[FFY00]

They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

I completely agree that that's a mistake

There were very good reasons for that, btw. And deprecating is different than removing, currently there's no plan to drop support for invoking setup.py directly.

If meson had a command that'd yield the exact same result as pip install . --no-build-isolation then I'd say we could use that and there'd be no need to go through a wheel. But there's no such command (yet, at least).

I essentially proposed this in https://github.com/mesonbuild/meson/issues/11462.

[FFY00]

importlib.machinery.EXTENSION_SUFFIXES here is only as a safety check. We can remove it. But I would be very surprised if there are no other things that break.

I think the best action here is to skip the check when cross compiling.

[eli-schwartz]

There were very good reasons for that, btw. And deprecating is different than removing, currently there's no plan to drop support for invoking setup.py directly.

I really don't think there was. If my PR to enhance python -m setuptools.launch (an existing functionality) had been accepted, it could have been built upon to provide:

• reading of setup.cfg setup_requires, before evaluating setup.py
• reading of pyproject.toml build-system.requires, before evaluating setup.py
• reimplementing setup.py install to be based on setup.py bdist_wheel instead of setup.py install_egg_info, or even just copying it to setup.py bdist_install.

(The first two of these are the classic issues brought up for why a setuptools command line is inherently bad, and they're the easiest ones to solve, too. In general, I agree that they're all worth solving. Ultimately they ended up being solved via deprecation, rather than via making it work.)

Since there was active disinterest in this, I stopped arguing. The current methods have viable handling. There were unexplored alternative options, but it is what it is, at this point. :)

[FFY00]

That issue has already been discussed at length in the proper places, so I am not gonna repeat that discussion here. Your proposal had nothing to do with avoiding the deprecation of setup.py calls.

[eli-schwartz]

I... don't think I ever did say any such thing???

I did say that I didn't bother attempting to make any additional proposals, since the one I did make was rejected on the grounds that the entire topic of setup.py calls was deprecated and there was no interest in un-deprecating it.

[FFY00]

As a reply to me saying that direct setup.py invocations were deprecated for good reasons, you said you didn't think so and then mentioned your proposal, so I read it as you were trying to say it was somehow meant to fix things. But I guess it was meant to say CLI support could be kept? Anyway, the setuptools maintainers decided that invoking setup.py was deprecated, and that they didn't want to support a new CLI[^1] in favor of 3rd party PEP 517 frontends. I think this makes sense, because it'd be even more confusing to have yet another tool-specific CLI, while we are pushing for standardized tools, and that CLI would only be available on newer setuptools versions, while PEP 517 just work everywhere.

This isn't very relevant here anyway.

[^1]: Your proposal was a new CLI that used setuptools.launch internally, and setuptools.launch was never even meant to be used like that (see https://setuptools.pypa.io/en/latest/history.html#id1213), so yes, I am considering a new CLI, it'd be a new functionality.