[BUG]: Does return_value_policy::reference_internal apply keep_alive<0, 1>?

[riclarsson]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [x] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [x] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

return_value_policy::reference_internal does not seem to apply keep_alive<0, 1> despite documentation indicating that it does. This of course leads to unexpected behavior. Manually adding keep_alive<0, 1>() at the appropriate position fixes the problem.

There is a follow-up problem related to this that I am unable to reproduce in the example-code below but which might help be aware of if this leads to any code change. When using pure keep_alive<0, 1>() and not using return_value_policy::reference_internal, the value is properly kept alive as expected. However, the deletion of the value might itself cause python to call abort stating that some pointers that are freed were not allocated (malloc error). I am not able to reproduce this below, and I do not know why. I am still adding it here for additional context.

Reproducible example code

In C++, a class that does not return a reference to a value directly but indirectly:


#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

#include <ostream>
#include <sstream>
#include <variant>

using VariableType = std::variant<long int, double>;
std::ostream &operator<<(std::ostream &os, VariableType vt) {
  return std::visit([&](auto &&v) -> std::ostream & { return os << v; }, vt);
}

struct Foo {
  std::vector<VariableType> y;
  void push_back(VariableType z) { y.push_back(z); }
  VariableType &operator[](std::size_t i) { return y[i]; }
};

struct Bar {
  Foo &x;
  std::size_t i;
  VariableType &value() { return x[i]; }
};

namespace py = pybind11;

PYBIND11_MODULE(pybug, m) {
  py::class_<Foo>(m, "Foo")
      .def(py::init<>())
      .def("append", &Foo::push_back)
      .def(
          "value",
          [](Foo &foo, std::size_t i) {
            return Bar{foo, i};
          },
          py::return_value_policy::reference_internal);
  // add py::keep_alive<0, 1>() to not segfault

  py::class_<Bar>(m, "bar").def("__repr__", [](Bar &bar) {
    std::ostringstream os;
    os << bar.value();
    return os.str();
  });
}


In python, this now creates a segmentation fault:
```python
import pybug
foo = pybug.Foo()
for i in range(3001): foo.append(i)
bars = [foo.value(i) for i in range(3001)]
repr(bars)  # works
del foo
repr(bars)  # segfault