[BUG]: Crash if 2 modules have conflicting module-local types

[godlygeek]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [ ] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

When two PyBind11 modules define different, ABI-incompatible classes with the same name, it's possible to pass either one of them to a function expecting the other, resulting in memory unsafety and crashes.

Reproducible example code

testcat.cpp

#include <pybind11/pybind11.h>
#include <iostream>

namespace py = pybind11;

struct Pet {
    char *name;
    const char *getName() const { return name; }
};

Pet createPet() {
    return Pet { "cat" };
}

void printPetName(const Pet& pet) {
    std::cout << "pet name: " << pet.getName() << std::endl;
}

PYBIND11_MODULE(testcat, m) {
    m.def("create_pet", &createPet);
    m.def("print_pet_name", &printPetName);
    py::class_<Pet>(m, "Pet", py::module_local())
        .def("get_name", &Pet::getName);
}

testdog.cpp

#include <pybind11/pybind11.h>
#include <iostream>

namespace py = pybind11;

struct Pet {
    std::string name;
    std::string getName() const { return name; }
};

Pet createPet() {
    return Pet { "dog" };
}

void printPetName(const Pet& pet) {
    std::cout << "pet name: " << pet.getName() << std::endl;
}

PYBIND11_MODULE(testdog, m) {
    m.def("create_pet", &createPet);
    m.def("print_pet_name", &printPetName);
    py::class_<Pet>(m, "Pet", py::module_local())
        .def("get_name", &Pet::getName);
}

setup.py

from pybind11.setup_helpers import Pybind11Extension
from setuptools import setup

ext_modules = [
    Pybind11Extension("testdog", ["testdog.cpp"]),
    Pybind11Extension("testcat", ["testcat.cpp"]),
]

setup(name="pettest", ext_modules=ext_modules)

pyproject.toml

[build-system]
requires = ["setuptools", "wheel", "pybind11"]
build-backend = "setuptools.build_meta"

crash.py

import testcat
import testdog
testdog.print_pet_name(testcat.create_pet())

Results

Dropping all of those files into a directory and then running:

$ pip install .
$ python crash.py

results in a segmentation fault, due to incorrectly treating a Pet from testcat.so as though it's a Pet from testdog.so, rather than a distinct type that happens to have the same name.

This appears to have been an intentional feature introduced by #1007, but application of this feature across different modules built at different times against different (potentially ABI incompatible) libraries can result in memory unsafety and undefined behavior.

[godlygeek]

Note also that because of PyBind11's hidden symbol visibility, despite having the same name these are distinct types as far as the loader is concerned, and therefore there's no risk of an ODR violation resulting from an UNDEF in one shared library being satisfied by an ABI-incompatible object exported by the other shared library.