Implement `#[init]` method attribute in `#[pymethods]`

[ahlinc]

This allows to control objects initialization flow in the Rust code in case of inheritance from native Python types.

A simple example, you implement a class inherited from PyDict but want to build a custom constructor interface. With just the #[new] constructor it's not possible and the base PyDict __init__ method will be called too with all arguments that were passed to the constructor as a standard objects initialization behavior.

[ahlinc]

It seems that most of codecov/patch complains are unrelated to this PR changes.

[mbway]

btw I also have basic support for defining __init__ in my PR because it is required for defining a metaclass. I haven't compared the implementations but mine has the limitation of not supporting inheritance properly so this is probably better.

https://github.com/PyO3/pyo3/pull/4678#issuecomment-2495184649

There is a limitation with extending PyType that tp_new cannot be set so only tp_init is available. My workaroud for now is to initialize to Default::default() before calling the user's init function but this doesn't work with multi-level inheritance (currently I catch this case with an assert). I decided not to invest more into this because it might be entirely scrapped in favour of re-purposing #[new] in the case of a metaclass but that's open to discussion. I thought the init approach would at least be simpler to implement for my proof of concept.

[Icxolu]

I've refactored this to current main, as well as #5551, for now I'm switching this to draft until that lands. While I was at it, I also implemented most of the review feedback from above.

[Icxolu]

With #5551 landed, I think this is now also in a reviewable state again.

[Icxolu]

Not sure whats up with fmt, it works locally. These line numbers don't exist in that file and Any is clearly define in the top of it... 🤔 (Probably I'm just blind 🤷 )

[davidhewitt]

Try merging with main? Probably operating on a merge commit 🤔

[Icxolu]

That did the trick 🙃

[Icxolu]

Looks like the PyPy failure is real. I've managed to reproduce it on linux (wsl) but not windows.

[Icxolu]

Looks like some kind of memory corruption that occurs when we try to run the base initializer:

self_.py_super()?.call_method("__init__", args.to_owned(), kwargs)?;

Not sure why tho 🤔

[davidhewitt]

Hmm, I will try to help investigate ASAP, probably Wednesday. This is probably good reason to delay #5646 just in case it needs a patch to the existing code.

[davidhewitt]

Hmm, I will try to help investigate ASAP, probably Wednesday. This is probably good reason to delay #5646 just in case it needs a patch to the existing code.

Not forgotten about this, have had some family stuff come up this week which is keeping me away from the linux desktop.

[Icxolu]

No worries. I tried to debug a bit yesterday, but without much success. At least the slf, args and kwags pointers look still fine when we call into Python. Might be something inside the object(s) that gets corrupted.

[davidhewitt]

From examining valgrind output, it seems to me that we have an issue where the PyPy GC is attempting to interact with the dict subclass after it's been deallocated. Presumably calling __init__ on the dict subclass is causing the PyPy GC to track it in some form.

I can replace the use of py_super with a handwritten call and I still get the crash, so it is definitely not related to py_super itself:

use pyo3::PyTypeInfo;

let mut full_args = vec![self_.clone().into_any()];
for arg in args {
    full_args.push(arg);
}

let args = PyTuple::new(self_.py(), full_args)?;
PyDict::type_object(self_.py()).call_method("__init__", args, kwargs)?;

I can also trigger the crash more reliably by forcing gc collection after deleting the dict subclass:

$ git diff tests
diff --git a/pytests/tests/test_pyclasses.py b/pytests/tests/test_pyclasses.py
index e2ef59ef0..2db6db71b 100644
--- a/pytests/tests/test_pyclasses.py
+++ b/pytests/tests/test_pyclasses.py
@@ -1,3 +1,4 @@
+import gc
 import platform
 import sys
 from typing import Type
@@ -177,3 +178,9 @@ def test_class_init_method():

     d = SubClassWithInit({"a": 1}, b=2)
     assert d == {"__init__": True, "a": 1, "b": 2}
+
+    del d
+    for i in range(10):
+        gc.collect(0)
+        gc.collect(1)
+        gc.collect(2)

Not yet sure whether the issue is in our implementation of classes on PyPy, or an upstream PyPy issue. I'll try to minimise the repro.

[davidhewitt]

#5653 fixes 🎉

[Icxolu]

💯 Thanks for investigating! I suspected it had something to do with the subclassing, but only manifested itself due to the new __init__ call. I'll rebase this once #5653 lands.

[Icxolu]

Woohoo 🎉 all green ✅