Allow defining python extension types

[dgrunwald]

Basically https://docs.python.org/2/extending/newtypes.html, but from safe Rust code.

I'm currently working on this.

py_class! macro support for:

• [ ] inheritance
• [X] data members
• [ ] DST data members
• [X] instance methods
• [x] class methods
• [x] static methods
• [x] static class variables
• [X] def __new__
• [ ] other special methods
• [ ] properties
• [ ] docstrings
• [x] GC integration
• [ ] pickle support

[novocaine]

:+1:

[dgrunwald]

I've completely changed my approach on this.

PyRustTypeBuilder (now: rustobject::TypeBuilder) is nice for allowing the dynamic creation of types, but it isn't really usable in its current state given that you can't use py_method!() with a closure (python doesn't allow passing any additional state around to the callback).

The new approach is the super-powerful py_class!() macro, which is much more user-friendly (no need for the user to deal with storing a PyRustType<..> instance). It's currently backed by rustobject::TypeBuilder, but could possibly be changed to create static types (as one would use in a hand-written C extension).

Given the ability to create static types using py_class!(), it's theoretically possible to implement custom method descriptors that allow passing additional state around, thus allowing the creation of true dynamic types using Rust closures for callbacks. Not sure if I want that to be a part of this library; for now, consider this issue to be only about py_class!(), and TypeBuilder deprecated (only for internal use in macros).

[thanatos]

Well, I wish I'd see this sooner. I've been struggling with PyRustTypeBuilder for a few days ago, and was running into the closure thing as well. I've switched to py_class!; it's much more useful, especially since you have examples. (I was going to file a bug for that, but then found this bug.)

It seems like py_class!() forces the Rust type name to be the same as the Python one? (I'd kinda like to prefix Py in Rust, as it is essentially going to wrap a Rust type by the same name, and having exactly the same name causes it to shadow.)

Now all I need is __next__. Looked briefly, and I think py_class_impl.py is where it needs to go, but everything is unimplemented() so it's hard to find an example, I guess.

[dgrunwald]

Yes, the macro forces the Rust type name to be the same as the Python class name. Do you have any suggestion for a syntax that allows the user to specify the name separately? Alternatively, you could use a nested Rust module to avoid the naming conflict.

I just implemented __iter__ and __next__ in d68e6643279ac62a884800cdb3116cc246ed2935.

[thanatos]

Sorry it's taken a while to get back to this; first, thanks so much for implementing those! I ran into a bit of trouble, all of it in my code, but perhaps it might inform your examples or documentation.

First, I found myself doing,

    def __next__(&self) -> PyResult<PyObject> {

which doesn't work; I need a PyResult<Option<PyObject>>; rustc will tell you this directly, but it wasn't evident to me as to why I needed that as a return type; I was expecting to need to write __next__ like I would in Python, i.e., I would need to raise StopIteration; the piece of info I was missing was IterNextResultConverter, which magically transforms that Option<T> into either a return of T or a raise StopIteration; i.e., a cool translation between Rust and Python. Just didn't know it was there.

The next mistake I made was,

    def __next__(&self) -> PyResult<Option<PyObject>> {
        Ok(Some(py.True()))
    }

My naïve expectation here was that the PyBool would get transformed into a PyObject automatically. When it didn't, I ended up looking for a method to do that, but didn't find one at first. The end result is even simpler than I expected … I can just return …Option<PyBool>, and apparently your framework magics the result into a Python object? (It looks like IterNextResultConverter might do this in its Some(val) branch?) (I did later find .into_object() in PythonObject, which also works.)

At any rate, I made an iterator that returned an infinite stream of Nones, and then an infinite stream of bools, so I'm thinking the remainder of the work should be easy to get it to do something more practical.

[dgrunwald]

The py_class! documentation lists the special methods and their expected signatures.

[thanatos]

Oh perfect! Not sure why I didn't run across that in my debugging yesterday. (Though maybe it was for the better — I learned a lot about how to inspect Rust macros…)

[thanatos]

Oh, not sure how much this matters, but:

<cpython macros>:172:1: 172:9 error: macro undefined: 'py_error!'
<cpython macros>:172 py_error ! { "__aenter__ is not supported by py_class! yet." } } ; {
                     ^~~~~~~~

I got this too w/ __next__ before you implemented it; the py_error! macro doesn't seem to exist / be findable. It wasn't really a big deal, since the error that results is still perfectly clear.

[Alphare]

I see in py_class_impl.py that __init__ is not supported and __setattr__ is unimplemented. Is there any way of currently creating a class with instance attributes? If no, are there any plans for implementing those features ? If I could be of any help, let me know.

[dgrunwald]

__init__ interacts badly with Rust's requirement that fields are initialized. Particularly once you throw inheritance into the mix -- if a derived class forgets to call super(self).__init__(), how would the rust data members get initialized? To allow constructing instances of a py_class from Rust, use __new__ instead.

__setattr__ could be supported; just requires someone putting in the work. The underlying slot has slightly different semantics as the python special method: tp_getattro corresponds to __getattribute__ not __getattr__; and tp_setattro corresponds to __setattr__ and __delattr__ together. This makes their support not so straightforward, as the macro needs to generate a non-trivial wrapper function.

[dgrunwald]

Though if you only need __setattr__, that shouldn't be so difficult. The main problem will be generating a single wrapper that can dispatch to either __setattr__ or __delattr__ when both are defined. That requires a bit of logic to know which kind of wrapper to generate depending on the defined special methods; which would be simple if we were using procedural macros, but py_class! predates those. Now macro_rules! is turing-complete and perfectly capable of such logic, but it's highly annoying to work with for such complex tasks.