Guide for how to build extensions in rust

[novocaine]

rust-cpython can be used to build extensions as well as to embed python in a rust process, so we should document that and make it easy for people.

To that goal, I have started on some distutils helpers for building and packaging extensions written as rust crates over at https://github.com/novocaine/rust-python-ext. I have used the example hello world code from rust-cpython as a start.

These helpers don't depend on rust-cpython, but I'd like to know your thoughts on either just merging them into this repo (so rust-cpython is 'batteries included'), or maybe the example in this repo could just depend and/or recommend them. I don't mind either way.

[ghost]

Oh, interesting! I didn't know whether embedding was easily supported by this library. This looks delicious.

I had already started a little collections of rust-embedded-in-python examples at https://github.com/hsoft/rust-python-embed but it doesn't use rust-cpython and thus is much less elegant than your example.

Hello world examples are nice, but they don't show tricky memory management problems. One day when time allows me, I'll add more examples to your repo.

[novocaine]

Looks like we share a common interest :+1:

The aim of rust-python-ext is that the rust extension will be magically built and installed when the user types python setup.py install just as it is for a c-ext or cython script. Assuming the system has rust installed.

It solves many tricky packaging issues like making sure everything is built against the right python, ability to co-exist with cython and c-exts, installation of the .so to the right place, respecting custom platlib, no reliance on third party build scripts, etc etc.

rust-cpython obviously needs more worked examples - so yes please! It'd be great if you could port your example problems to rust-cpython. And even nicer if you use rust-python-ext (when it's not vapourware) :D

[ghost]

My immediate problem is that I'm not setup for rust nightly (which this project seems to require). I need to find a workflow for switching between stable/nightly. Soon...

[dgrunwald]

I'm not familiar with setup.py. Feel free to submit examples as pull requests.

[ihrwein]

@hsoft You can use multirust for switching.

[Ronaldho80]

Hello,

I am trying to build python extensions with cpython in Rust 1.10 nightly. From what I found in the cpython repository, it should be pretty straight forward. I copied the example hello.rs into a dedicated rust project and compiled it with cargo build to a lib with Cargo.toml:

[package]
name = "hello"
version = "0.1.0"
authors = ["Ronny Herzog <[email protected]>"]

[dependencies]
cpython = { git = "https://github.com/dgrunwald/rust-cpython.git" }

[lib]
crate-type = ["dylib"]

However, Python (2.7) is giving me an ImportError: dynamic module does not define init function (inithello).

I am looking urgently for a good description of how to generated a python extension with cpython.

[cmyr]

@Ronaldho80 I appreciate this may be too late to be helpful, but after a bit of experimenting I have a sample project that builds a lib (using stable rust) which you can look at: https://github.com/cmyr/rust2py.

@dgrunwald this seems like something people generally want to do, and I'd be happy to add something to the readme if you'd like? The hello.rs extension was helpful but having to build through make posed some problems. I think some sort of barebones skeleton example-project like the one in my repo might be a useful resource, but not really sure how well it would fit here. Anyway this is very much toes-in-the-water time for me, happy to have this working though, thanks for all your work. 😄

[dgrunwald]

@cmyr: Feel free to submit a pull request updating the readme.

A few remarks regarding your example code:

• Don't mix tabs and spaces, it makes the code look ugly on github.
• extract already returns a PyErr on failure, so there's no need to construct your own unless you want to use a custom error message
• py_fn! can do the extraction for you, e.g. you could just use py_fn!(py, fib(arg0: u64)) or py_fn!(py, reversed_words(text: &str)).

[cmyr]

@dgrunwald thanks for the feedback! didn't notice I had mixed whitespace, I'll take a look at my editor settings.

on extract: good to know. re: py_fn, this was definitely one of the things I had the most trouble with, conceptually: rust macros still feel a bit magical. Does this mean that I can declare my rust function implementation to take a rust type, instead of a PyObject? My reasoning here was that since there's no type-checking on the python end I want to be able to check invalid input and handle it appropriately, instead of having a panic elsewhere. (Also the only example code I could find was using either no args or PyTuple/*PyDict args).

Do you have any opinions on what best practices might be here?

[Ronaldho80]

@cmyr thanks for coming back to my issue. I finally could manage on my own. However, I think many people would appreciate a nice tutorial and/or example.

[cmyr]

@Ronaldho80 There's a simple example now on the main readme. Is there anything you think is missing? I'm only really starting to actually work with this code now, but I'll take notes of my pain-points and maybe put something more comprehensive together.

[Ronaldho80]

The problem was that I was using Python2.7. But I still lack knowledge of some details in the Cargo.toml compiling description. In particular, I did not know how to import cpython for Python2.7. I ended up with cloning the repository and removing some lines in the cpython Cargo.toml. However, this seems not to be idiomic. I miss a description on how to simply switch to Python2.7. Thx, Ronny

[ihrwein]

@Ronaldho80 I suppose you can switch to python2 by telling cargo to use the python27-sys feature. I'd try this command (though I don't have access to cargo right now):

cargo build --no-default-features --features python27-sys

[expobrain]

Here an example of how to do it in Cargo.toml:

[dependencies.cpython]
git = "https://github.com/dgrunwald/rust-cpython.git"
default-features = false
features = ["python27-sys"]

[Ronaldho80]

Cool, thx!