Rust extension given invalid name by meson-python

[medley56]

Link to reproducible example: https://github.com/medley56/python_rust

When using meson-python to package a Rust extension, I get the following error when running pip install .

../meson.build:10:14: ERROR: Rust crate python_rust_lib.cpython-312-darwin type dylib does not allow spaces, periods or dashes in the library name due to a limitation of rustc. Replace them with underscores, for example

The project works when using maturin as a build backend so I'm reasonably sure it's not a problem with the actual extension. I believe the problem is in meson-python or in my configuration of meson-python, causing the backend to build the rust extension with an invalid name.

Platform: MacOS Sequioia 15.3

Documentation I've consulted: Tutorial: https://mesonbuild.com/meson-python/tutorials/introduction.html#tutorial meson docs for extension_module: https://mesonbuild.com/Python-module.html#extension_module PyO3 docs (uses maturin): https://pyo3.rs/v0.23.4/module.html

[rgommers]

Thanks for the bug report and reproducer @medley56! We should make that work - I think the reason you're running into it is that you're the first one to try. Rust support in Meson is pretty recent, and none of the maintainers of meson-python are regular Rust users I believe.

One thing that jumps out when browsing the code is that this line in meson.build:

py = import('python').find_installation()

should be:

py = import('python').find_installation(pure: false)

That may not be the problem though, I'll try to test as soon as possible.

[rgommers]

Let's call this a feature request rather than a bug. Maturin does the following (from https://www.maturin.rs/bindings#pyo3): "maturin automatically detects pyo3 bindings when it's added as a dependency in Cargo.toml."

Neither Meson nor meson-python does anything like this. Using py.extension() on a Rust source file needs to do either something like what Maturin does (adding a pyo3 dependency and invoking the right machinery), or the user must do this explicitly in meson.build instead of feeding a single .rs file to py.extension.

I did a bit of digging in the Meson issue tracker, but can't find any relevant discussions - it may be that no one has worked on this yet.

[rgommers]

In meson-python itself we should probably start by adding some test cases for packaging projects that are known to work with Meson as wheels, just to make sure that that all works (it should).

[medley56]

Thanks for the attention to this! Looking forward to being able to use meson to build a Rust extension. It sounds like you have an idea for a workaround:

or the user must do this explicitly in meson.build instead of feeding a single .rs file to py.extension

Is that something simple/useful or would you rather just call this unsupported until a better solution can be implemented? For my part, I'm happy to just use maturin for now.

[greglucas]

It looks like if the crate is declared as cdylib, then we need to specify the rust_abi in the extension module.

rust_ext = py.extension_module(
    'python_rust_lib',  # Output name
    'src/lib.rs',  # Rust source
    install: true,
    rust_abi: 'c'
)

However, it appears that building the extension module still doesn't follow the crate dependencies and install those (it can't find pyo3 when compiling). Do you need to manually declare a dependency on other crates?

[eli-schwartz]

Meson can have a rust target depend on a crate: https://mesonbuild.com/Wrap-dependency-system-manual.html#cargo-wraps

dependencies: [dependency('foo-rs')]

and a cargo subproject dependency will automatically parse Cargo.toml and depend on its dependencies. But you still need to define your direct dependencies, just like you do with cargo (but with cargo, you define them in Cargo.toml instead of meson.build).

FTR: I don't really have experience with meson's rust support, that is being primarily driven by maintainers from the Gnome and Mesa ecosystems. I don't know offhand about anyone doing python module experiments with it, the experiments I've seen have involved replacing C/C++ libraries and executables. In theory, pyo3 extensions are "just" libraries that happen to utilize libpython, so it should be more or less the same modulo usage of the pyo3 crate itself.

[dnicolodi]

I've tried to put together an example project building with Meson a Python module using PyO3. The issue with the cdylib being given a Rust invalid name can be worked around compiling a Rust static lib and then linking it into a python module whose sources are an empty C file (at least Meson does not complain about this, I'm not sure if it actually works when linking). The second ingredient is a Cargo.lock that lists PyO3 and all transitive dependencies. Then Meson is able to resolve dependency('pyo3-0.25-rs'), generate wrap files for all required crates and start building. However, the build does not go very far: one of the PyO3 dependencies uses some proc_macro dependency and AFAICT Meson is not yet able to make sense of that. I don't know much about Rust (yet) and I don't know where to go from here.

I think refusing the name for the cdylib is a Meson bug and someone more familiar with Rust should probably have a look at this. I had a quick look at how maturin compiles Python extensions and AFAICT it directly instructs cargo to compile a cdylib with a name equivalent to the one the Meson's python module uses.

Paging @xclaesse, @dcbaker, and @bonzini for a consult.

[ngoldbaum]

I'm a PyO3 maintainer and am very interested in this, as it will eventually unblock adding Rust code to scientific python projects.

I discovered this issue while talking with @nrposner on the PyO3 discord, they also expressed interest in helping out.

@davidhewitt might also be able to answer questions from the Rust build side.

This is looking a little further ahead, I think eventually the main PyO3 repo could grow a meson-python example to go along with the maturin and setuptools-rust examples we have already.

[bonzini]

I hope that Meson 1.11 will be able to parse Cargo.toml from the toplevel project directory and use that to configure all the dependent crates (as long as they don't have a build.rs, at least, otherwise it won't be totally automatic).

Based on that, which is currently drafted at mesonbuild/meson#15223, one should be able to add the right keyword arguments to py.extension_module(), and it will link to pyo3.

rust_abi: 'c' however could be added by extension_module already now, since a Rust-ABI Python extension module makes no sense.

[dnicolodi]

Thanks for the interest. I don't think there is anything that needs to be done on the meson-python side of things. However, as pointed out by @bonzini there may be tweaks required in the Meson's python module. PyO3 seems to have quite a few crate dependencies. However, I believe, it should be possible to write a simple Python extension in Rust calling directly into the Python C API. If someone could provide that, it would be easy to add it as a test for meson-python and check what needs adjusting.

[eli-schwartz]

Note that there are some things which could be investigated in parallel to make things easier. Not familiar with the pyo3 codebase at all, but a common issue with rust crates in meson is:

• rust crates that use C/C++ DFOO=bar" macros, using the rust-native spelling env!`.

Rustc does have a -D option (they call it --set-env) but it is nightly-only and there seem to be no plans to actually stabilize it. Even the nightly flag is basically new. The original design of rustc required the build system generator invoking rustc (cargo, or ninja in the case of meson) to set environment variables instead, which ninja doesn't support. IIRC there was some drama about how C's implementation of source code configuration via tuneable definitions occurs before lexing and therefore command line arguments are bad, and the idea has never fully left the devs.

In general it's better to generate the source files instead, which meson can handle via structured_sources()

[bonzini]

PyO3 doesn't have many dependencies but it has a large build.rs. the main issue there will be to figure out how much of it duplicates the Meson Python module, and fill in the blanks.

Probably there would be a meson.build file that replaces build.rs when building PyO3 as a subproject, but that's getting a bit ahead of ourselves.