[BUG]: Overload resolution fails for derived types when single-argument overload is registered after vector overload

[dyollb]

Required prerequisites

• [x] Make sure you've read the documentation. Your issue may be addressed there.
• [x] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [ ] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

3.0.0rc2

Problem description

Description:

When exposing overloaded functions in pybind11 for both std::shared_ptr<Entity> and std::vector<std::shared_ptr<Entity>>, calling the function with a derived type like std::shared_ptr<TriangleMesh> (where TriangleMesh inherits from Entity) may fail at runtime:

TypeError: object of type 'XCoreModeling.TriangleMesh' has no len()

This occurs even when class inheritance and std::shared_ptr relationships are correctly registered.

Cause:

If the vector overload is registered before the single-object overload:

m.def("func", [](const std::vector<std::shared_ptr<Entity>>& v) { func(v); });
m.def("func", [](const std::shared_ptr<Entity>& e) { func(e); });

pybind11 may mistakenly match the vector version and attempt to treat the object as a container, resulting in a len() error.

Workaround:

Switching the order of registration ensures the correct overload is selected:

m.def("func", [](const std::shared_ptr<Entity>& e) { func(e); });
m.def("func", [](const std::vector<std::shared_ptr<Entity>>& v) { func(v); });

Recommendation:

Document or improve overload resolution behavior in pybind11 when dealing with ambiguous cases involving shared pointers and containers.

Reproducible example code

Is this a regression? Put the last known working version here if it is.

Not a regression

[GS-GOAT]

@dyollb The problem is due to how pybind11 resolves overloaded functions. the Two-Pass System in Pybind11 attempts to find the correct overload in two passes - It first tries to find an overload that matches the provided arguments exactly, without any implicit type conversions. If the first pass fails, it then tries to find an overload that can be called by performing implicit conversions on the arguments. Registration Order matters within each of these passes, pybind11 tries the overloads in the order you registered them in your C++ code - https://pybind11.readthedocs.io/en/stable/advanced/functions.html So if two overloads are equally viable (lets just say both require a conversion), the one that was defined first with .def() will be chosen.

I can raise a PR for this quick fix if its fine by you.

[dyollb]

Thanks for the explanation. Makes sense. Is your quick fix to add a comment in the documentation? Sounds good, thanks

[GS-GOAT]

For now adding a note in the documentation will help . For the long term fix , work can be made towards how resolution works. pybind11 would need a system for ranking the quality of a potential match. Ofcourse this would be a significant design trade-off. The root of this issue is that the type caster for std::vector is intentionally greedy. It's designed to accept any Python object that is iterable. This flexibility is useful in cases like passing a Python tuple or a generator to a function that expects a std::vector, but it causes ambiguity in this specific overload scenario.