Prepare pybind11 v3 update (with typing fixes)

[timohl]

Type

• [X] Bug fix (non-breaking change which fixes an issue): Fixes #
• [ ] New feature (non-breaking change which adds functionality). Resolves #
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

This PR prepares upgrading pybind11 to v3. Two days ago pybind11 v3.0.0rc1 was released as a release candidate for v3. This includes a lot of type hint improvements which are required for including stubs in the Open3D Python package (see #6917)

Checklist:

• [ ] I have run python util/check_style.py --apply to apply Open3D code style to my code.
• [ ] This PR changes Open3D behavior or adds new functionality.
  • [ ] Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is updated accordingly.
  • [ ] I have added or updated C++ and / or Python unit tests OR included test results (e.g. screenshots or numbers) here.
• [X] I will follow up and update the code if CI fails.
• [X] For fork PRs, I have selected Allow edits from maintainers.

Description

This PR already contains some type hint related fixes. The release candidate contains an upgrade guide. I will try to follow it and update this PR (probably this weekend).

TODOs:

• [ ] Fix deprecated features
• [ ] Consider using py::smart_holder over std::shared_ptr (docs)
• [ ] Consider using py::native_enum (docs)

[update-docs[bot]]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

[johnnynunez]

good @timohl i need it!

[timohl]

I changed this PR to just include the changes required to update to pybind11 v3.0.0 (leaving the typing fixes in the examples for another PR).

Some deprecated functions had to be replaced or refactored, FunctionDoc parsing and fileystem header needed small fixes. Those commits should not change functionality.

Pybind11v3 introduces py::native_enum and py::enum_ is deprecated now. I have updated all enum bindings. This is an important part to review, since it changes the behaviour. I changed most enums to the enum.Enum type. If the old binding used py::arithmetic(), I chose between 3 types:

• If the underlying enum uses bits -> enum.IntFlag which supports bit-wise operations, e.g., flag1 | flag2.
• If the enum requires integer operations -> enum.IntEnum, e.g. log verbosity needs comparison log_lvl >= VerbosityLevel.Warning
• Otherwise, I chose enum.Enum Here it would be helpful if somebody could look through the changed enums and see if the enum type makes sense.

Also, I am not quite sure what the impact on existing code is. Previously, comparison of enum values had to be done using == instead of is, but I am not sure if == is still ok now.

Furthermore, I changed the enums to use the docs argument of py::native_enum instead of manually assigning a function to the __doc__ attribute.

Some enums had conflicting values with .export_values() so I removed the export for those.

Next, I would like to change the holder to py::smart_holder for all classes using std::shared_ptr which is now recommended. There are 455 py:class_ in the code base and the bind_vectors have to be checked, as well. I will do that the next days.