Scaling PyO3 down with single-file Python extensions

[itamarst]

One nice thing about Cython is that you can have a single .pyx file, two lines in setup.py and one line in requirements.txt, and you're good. With PyO3, you need a Cargo.toml, configured a particular way (cdylib), and so it's just a bit more ceremony than Cython for simple cases.

Maturin is great for standalone libraries, but doesn't help simplify the case of adding a quick extension to an existing Python project.

To solve this, one could imagine single-file Rust extension modules, inspired by https://github.com/rust-lang/rfcs/blob/master/text/3424-cargo-script.md (and see the references to existing tools that predate the RFC, mostly for executables). You might then have:

setup.py
mypackage/
    __init__.py
    module.py
    _extension.rs

And the setup.py is modified to point at _extension.rs, much like you would with Cython, rather than pointing at Cargo.toml as it normally does:

from setuptools import setup
from setuptools_rust import Binding, RustSingleFileExtension

setup(
    name="mypackage",
    version="1.0",
    rust_extensions=[RustSingleFileExtension("mypackage._extension", binding=Binding.PyO3)],
    packages=["mypackage"],
    # rust extensions are not zip safe, just like C-extensions.
    zip_safe=False,
)

The _extension.rs might look like this:

//! ```cargo
//! [dependencies]
//! pyo3 = "0.19"
//! ```

use pyo3::prelude::*;

/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())
}

/// A Python module implemented in Rust.
#[pymodule]
fn _extension(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
    Ok(())
}

Once you start thinking about lock files this starts getting more complicated, they could be auto-generated as mypackage/_extension.rs.lock, and you'd need some way to update them... conceivably could outsource that to Cargo eventually if the feature becomes non-experimental and they are willing to add a new use case.

[itamarst]

As an alternative to the above syntax, the config part could be specified in setup.py as an argument to RustSingleFileExtension (or whatever it ends up being called) rather than in the Rust file.

[itamarst]

Some reasons this might be a bad idea:

1. Lack of reproducibility, if lock file support is not included. Adding lock file support increases scope.
2. Scaling up may become more complex since now you need to convert the current config to a Cargo.toml. This can likely be solved with tooling, but that's an increase in scope.
3. More likely to have errors (no syntax highlighting) and more difficult debugging of errors, in the Cargo.toml-equivalent config.

[adamreichold]

I think if one goes automatically generating a Cargo.lock, one could also implement this as automatically generating (or updating) a Cargo.toml file. The canonical path of src/lib.rs is only conventional IIRC and a Cargo.toml could change this to extension.rs.

[itamarst]

One alternative I just discovered to the proposal here is https://github.com/mityax/rustimport

I'll try playing around with it, and report what I've learned here.

[alex]

It seems to me that once the cargo-script RFC is stable, it'd be a much smaller amount of work to build this on top of that.

[itamarst]

So I played around with rustimport (https://pythonspeed.com/articles/easiest-rust-python/), and my feeling is it does quite well at simplifying prototyping. And there's maybe a use case for single-file Rust that is an extension you ship or distribute, but... it feels like prototyping is maybe the bigger use case.

So from my perspective it's probably not worth spending a lot of time on adding it to setuptools-rust, I'm happy to close this issue.

[itamarst]

Another alternative: https://www.maturin.rs/develop.html#import-hook