[BUG]: 'terminate called after throwing an instance of' in `def_buffer` method

[stellaraccident]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

It seems that if I throw an exception from a def_buffer method, this is not translated properly to a Python exception and instead terminates the program with something like:

terminate called after throwing an instance of 'std::invalid_argument'
  what():  unimplemented array format.
Aborted

Reproducible example code

I believe the following minimal repro causes this but I have not yet built it separately/verified outside of my project.

struct Foobar;
py::class_<Foobar>(m, "Foobar")
  .def(py::init<>())
  .def_buffer([](Foobar &) {
    throw std::invalid_argument("boom");
  });

import numpy as np
import my_module
np.array(my_module.Foobar())

[Skylion007]

We probably should add bindings to BufferError as well as that might make more sense to raise an error here.

[rwgk]

terminate called after throwing an instance of 'std::invalid_argument'

Did someone already analyze why this happens? The usual suspects:

• Exception thrown from a destructor?
• Exception thrown while a previous exception is still unwinding?

(https://en.cppreference.com/w/cpp/error/terminate has a lot more details.)

Maybe if we understand the code path, we can think of a way to avoid the std::terminate?

[jbms]

The bf_getbuffer handler installed by pybind11 just calls the user-defined function without catching and translating any exceptions.

https://github.com/pybind/pybind11/blob/f791dc8648e1f6ec33f402d679b6b116a76d4e1b/include/pybind11/detail/class.h#L551

I have seen quite a few cases in pybind11 where c++ exceptions are allowed to propagate out of Python C API callbacks, though I think most of those cases are internal errors that aren't too likely to occur.

[Skylion007]

@jbms I suspect that this was the cause. Also, just to nit pick the error thrown really should be a BufferError since we have exception bindings for it. ;)

[Skylion007]

The exception is thrown here outside the scope of any exception_translators and within a new: https://github.com/pybind/pybind11/blob/e7c9753f1d35061d137ac3ce561e94a7407e5583/include/pybind11/pybind11.h#L1460

[Skylion007]

This is apparently a duplicate of https://github.com/pybind/pybind11/issues/2764#issue-776844461

[rwgk]

Do you think

try {
  return new buffer_info(((capture *) ptr)->func(caster));
 } catch (...) {
  // translate exception
  return nullptr;
}

will solve this?

[jbms]

It would be nice to factor out the exception translation logic from cpp_function so that it can be used in other places within pybind11, such as here, and also by user code directly when needed.

[rwgk]

It would be nice to factor out the exception translation logic from cpp_function

I think that's a great idea. @Skylion007, @henryiii, WDYT?

[Skylion007]

That definitely does sound useful.