[BUG]: C++ instance passed as pointer not marked as optional in function signature

[guillaumekln]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

When passing a C++ instance as a pointer, the default behaviour is to accept None as described in the documentation. Consider the following example:

py::class_<Dog>(m, "Dog").def(py::init<>());
m.def("bark",
      [](const Dog* dog) { return dog ? "woof!" : "(no dog)"; },
      py::arg("dog")=py::none());

The function signature in help(module) looks like this:

FUNCTIONS
    bark(...) method of builtins.PyCapsule instance
        bark(dog: module.Dog = None) -> str

but we would expect this signature instead:

FUNCTIONS
    bark(...) method of builtins.PyCapsule instance
        bark(dog: Optional[module.Dog] = None) -> str

I tested with pybind11 2.7.1.

Reproducible example code

No response

[henryiii]

Or, modern Python: dog: module.Dog | None. The problem with fixing/improving the type signatures to be properly MyPy compatible (which I'd love) is that that might increase the binary size a bit, and that's something pybind11 only does in dire circumstances. If there was a patch proved to not increase the binary sizes (a few bytes for the extra characters is okay), then it would probably be accepted. Otherwise, there likely will be a branch and a lot of coherent work to try to make this possible, and again at the end minimal increase will need to be shown. We don't produce valid NumPy types either (and that's been changing the last few NumPy versions, but hopefully is stable as of 1.21).

[guillaumekln]

As a workaround, using std::optional explicitly works as expected (the argument is marked as "Optional" in the function signature):

m.def("bark",
      [](const std::optional<Dog>& dog) { return dog ? "woof!" : "(no dog)"; },
      py::arg("dog")=py::none());

[tadeu]

Would the implementation be something like adding a PYBIND11_TYPE_CASTER with "Optional[" + ... + "]" around here? (Or possibly here and in other pointer types?)

[BenBE]

Similarly py::bytes* foo() should probably generate a signature of foo() -> Optional[bytes] or foo() -> bytes | None IMHO. Currently it produces plain foo() -> bytes which is misleading. Cf. https://github.com/pybind/pybind11/discussions/3377 for the context where I noticed this behavior.

[sarlinpe]

Is there a way to make this work with the new Optional type introduced in 2.13.0? Or any plan to fix this issue?

[VelocityRa]

Bump? It's been 3 years.

Pointers in C can be null. Seems rather a basic aspect to support. In C++ people often rely on that and intentionally choose pointers over references for that specific purpose. Why don't pybind types reflect this (even as an option)?

Not to mention std::function and other STL types that can be empty.