new feature: Improve automatic stub generation for python bindings

[chitralverma]

pyo3 stubgen. polars and duckdb use that AFAIK

https://crates.io/crates/pyo3-stub-gen

Originally posted by @chitralverma in https://github.com/apache/opendal/pull/6232#discussion_r2122572648

[Zheaoli]

for now, we have a template based generate code https://github.com/apache/opendal/tree/main/dev/src/generate

I'm not sure we can replace it or not

cc @Xuanwo

[Xuanwo]

for now, we have a template based generate code https://github.com/apache/opendal/tree/main/dev/src/generate

I'm not sure we can replace it or not

I think they serve different purposes. odev is used to generate python code, such as various service configurations. On the other hand, pyo3-stub-gen is used to generate python documentation from existing python code. That's a task we definitely don't want to handle ourselves.

I support using pyo3-stub-gen or any official solution provided by pyo3, if available.

[Zheaoli]

I support using pyo3-stub-gen or any official solution provided by pyo3, if available.

Same idea.

[chitralverma]

@Xuanwo once the 2 existing PRs get merged. let me take a shot at this, I have been meaning to try working with a rust/ python mix uv project for a while and this is an interesting thing for me.

if i get stuck i will let you all know for help.

Also, this current way has quite a poor IDE support, all the typing info is missed and errors show up with pyright and all. Will also do some linting/ CI improvements with this.

On the other hand, pyo3-stub-gen is used to generate python documentation from existing python code. That's a task we definitely don't want to handle ourselves.

from existing rust code, you mean (think)

[Xuanwo]

from existing rust code, you mean (think)

Oh yeah, you are right. Finally, we should migrate the template generated config to python bindings' rust side.

[Xuanwo]

By the way, I wish the stub generator could convert our Rust comments into Python API comments, so we could generate nice API documentation from them.

[chitralverma]

By the way, I wish the stub generator could convert our Rust comments into Python API comments, so we could generate nice API documentation from them.

i was just looking into this 😂 seems like there's no easy way for this, but let me see what can be done

[erickguan]

I had build some code to do this (in docs-service-website). I could build a service section in mkdocs website and type hints with docstrings. I was thinking having per-service config classes will be a nice feature to have firstly before getting more documentation from opendal-core.

[chitralverma]

By the way, I wish the stub generator could convert our Rust comments into Python API comments, so we could generate nice API documentation from them.

Resolved via https://github.com/apache/opendal/pull/6690 https://github.com/apache/opendal/pull/6703 https://github.com/apache/opendal/pull/6720 and https://github.com/apache/opendal/pull/6729

I was thinking having per-service config classes will be a nice feature to have firstly before getting more documentation from opendal-core. I could build a service section in mkdocs website and type hints with docstrings.

Resolved via https://github.com/apache/opendal/pull/6729