[BUG]: make_iterator causes runtime error in second scoped_interpreter

[jasjuang]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

This issue is similar to #2101, the reported minimal example problem in #2101 is fixed in the latest master because of #3744. However, the below minimal example will produce the error

1
2
3
terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  RuntimeError: instance allocation failed: new instance has no pybind11-registered base types

At:
  <string>(4): <module>

Aborted (core dumped)

I can reproduce this problem in Ubuntu and macOS.

The real use case is that each gtest TEST block will have its own py::scoped_interpreter, and this bug is preventing me from writing for i in testclass: more than once, but I would like to write for i in testclass: in multiple different TEST blocks.

Reproducible example code

CMakeLists.txt

cmake_minimum_required(VERSION 3.20)
project(pybindsample)

set(pybind11_DIR "/home/jasjuang/install/share/cmake/pybind11")
find_package(pybind11 REQUIRED)

pybind11_add_module(${PROJECT_NAME} pybindsample.cpp)
target_compile_features(${PROJECT_NAME} PUBLIC cxx_std_17)

add_executable(pybindsample_test pybindsample_test.cpp)
target_link_libraries(pybindsample_test pybind11::module pybind11::embed)

pybindsample.cpp

#include "pybind11/pybind11.h"
#include "pybind11/stl.h"

namespace py = pybind11;

class TestClass {
public:
  TestClass() = default;
  std::vector<int>::iterator begin() noexcept { return vec.begin(); }
  std::vector<int>::iterator end() noexcept { return vec.end(); }

private:
  std::vector<int> vec = {1, 2, 3};
};

PYBIND11_MODULE(pybindsample, m) {
  py::class_<TestClass>(m, "TestClass")
      .def(py::init<>())
      .def(
          "__iter__",
          [](TestClass &t) { return py::make_iterator(t.begin(), t.end()); },
          py::keep_alive<0, 1>());
}

pybindsample_test.cpp

#include "pybind11/embed.h"
#include "pybind11/pybind11.h"
#include "pybind11/stl.h"

namespace py = pybind11;

int main() {
  {
    py::scoped_interpreter g;
    py::eval<py::eval_statements>("import pybindsample\n"
                                  "testclass=pybindsample.TestClass()\n"
                                  "for i in testclass:\n"
                                  "    print(i)\n");
  }
  {
    py::scoped_interpreter g;
    py::eval<py::eval_statements>("import pybindsample\n"
                                  "testclass=pybindsample.TestClass()\n"
                                  "for i in testclass:\n"
                                  "    print(i)\n");
  }

  return 0;
}

[Skylion007]

Any thoughts @StarQTius?

[StarQTius]

The following snippet results in the expected behavior on my machine:

#include "pybind11/embed.h"
#include "pybind11/pybind11.h"
#include "pybind11/stl.h"

namespace py = pybind11;

class TestClass {
public:
  TestClass() = default;
  std::vector<int>::iterator begin() noexcept { return vec.begin(); }
  std::vector<int>::iterator end() noexcept { return vec.end(); }

private:
  std::vector<int> vec = {1, 2, 3};
};

PYBIND11_EMBEDDED_MODULE(pybindsample, m) {
  py::class_<TestClass>(m, "TestClass")
      .def(py::init<>())
      .def(
          "__iter__",
          [](TestClass &t) { return py::make_iterator(t.begin(), t.end()); },
          py::keep_alive<0, 1>());
}

int main() {
  {
    py::scoped_interpreter g;
    py::eval<py::eval_statements>("import pybindsample\n"
                                  "testclass=pybindsample.TestClass()\n"
                                  "for i in testclass:\n"
                                  "    print(i)\n");
  }
  {
    py::scoped_interpreter g;
    py::eval<py::eval_statements>("import pybindsample\n"
                                  "testclass=pybindsample.TestClass()\n"
                                  "for i in testclass:\n"
                                  "    print(i)\n");
  }

  return 0;
}

So it has to do with the dynamic linkage or maybe EMBEDDED_MODULES do some cleanup that MODULES don't I believe. I've checked with nm whether the local internals were duplicated, but it doesn't seem so.

[jasjuang]

@StarQTius I can also confirm that PYBIND11_EMBEDDED_MODULE works as intended but not PYBIND11_MODULE.

[StarQTius]

Well, I might have been wrong when I said locals were not duplicated. This is the address of the variable containing the internals when get_local_internals is called during the first interpreter finalization (in other word, this is when the fix of #3744 kicks in):

Breakpoint 1, pybind11::detail::get_local_internals () at /home/paulin/Desktop/test/pybind11/include/pybind11/detail/internals.h:515
515	    return locals;
(gdb) p &locals
$84 = (pybind11::detail::local_internals *) 0x55555559dd80 <pybind11::detail::get_local_internals()::locals>
(gdb) bt
#0  pybind11::detail::get_local_internals () at /home/paulin/Desktop/test/pybind11/include/pybind11/detail/internals.h:515
#1  0x000055555556a929 in pybind11::finalize_interpreter () at /home/paulin/Desktop/test/pybind11/include/pybind11/embed.h:203
#2  0x000055555556aa6e in pybind11::scoped_interpreter::~scoped_interpreter (this=0x7fffffffdd97, __in_chrg=<optimized out>)
    at /home/paulin/Desktop/test/pybind11/include/pybind11/embed.h:245
#3  0x000055555555b80e in main () at /home/paulin/Desktop/test/pybindsample_test.cpp:14

And this is what GDB displays when the breakpoint is hit twice more:

Breakpoint 1, pybind11::detail::get_local_internals () at /home/paulin/Desktop/test/pybind11/include/pybind11/detail/internals.h:515
515	    return locals;
(gdb) p &locals
$86 = (pybind11::detail::local_internals *) 0x7ffff6cc99a0 <pybind11::detail::get_local_internals()::locals>
(gdb) bt
#0  pybind11::detail::get_local_internals () at /home/paulin/Desktop/test/pybind11/include/pybind11/detail/internals.h:515
#1  0x00007ffff6c79788 in pybind11::detail::get_local_type_info (tp=...)
    at /home/paulin/Desktop/test/pybind11/include/pybind11/detail/type_caster_base.h:197

...

#55 0x000055555556f735 in pybind11::eval<(pybind11::eval_mode)2, 89ul> (s=..., global=..., local=...)
    at /home/paulin/Desktop/test/pybind11/include/pybind11/eval.h:85
#56 0x000055555555b87d in main () at /home/paulin/Desktop/test/pybindsample_test.cpp:17

It seems like the locals are internally linked. Is it the intended behavior ?

[Skylion007]

@StarQTius I don't think we have any defined behavior for the locals between interpreters (which is partially what caused these bugs). So whichever resolves these bugs. I don't see any reason why they should be linked unless there is some global configuration state that should be shared between them.

[StarQTius]

Then I think the best solution would be to register a callback in PyModuleDef::m_free which clear the locals, and keep one local_internals instance by shared library. In that case, EMBEDDED_MODULES would share the same locals, though I don't know if it will really be an issue (it might if the user try to register the same type in two different embedded modules I guess).

[ssooffiiaannee]

Could Anyone provide the compilation command, I'm getting a segmentation fault, I'm new to this.

[jasjuang]

@ssooffiiaannee

git clone https://github.com/pybind/pybind11
cd pybind11 && mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/home/jasjuang/install/ ..
make -j install
cd
mkdir sample && cd sample
<create CMakeLists.txt, pybindsample.cpp pybindsample_test.cpp>
mkdir build && cd build
cmake ..
make -j
./pybindsample_test