Issue binding protobuf RepeatedField and RepeatedPtrField

[cjgdev]

OS: Windows 10 Compiler: MSVC 2017 x64 Python: Python 3.6.4 (x64)

I'm trying to bind repeated fields generated by google protocol buffers. The binding code uses a custom type_caster to expose the RepeatedField and RepeatedPtrField types used by the automatically generated classes.

Here's a minimal protobuf definition that uses repeated fields. There's nothing particularly complex or in any way bespoke happening here.

syntax = "proto3";

message Value {
    repeated double data = 1;
}

message ValuesResponse {
    repeated Value values = 1;
}

To expose the repeated fields with pybind11, a specialism of the pybind11::type_caster is defined, and for convenience we create a specialism of the bundled pybind11::list_caster, which already does what we need.

namespace pb = google::protobuf;

namespace pybind11 { namespace detail {
	template <typename Type>
	class type_caster<pb::RepeatedField<Type>> : public list_caster<pb::RepeatedField<Type>, Type> {};

	template <typename Type>
	class type_caster<pb::RepeatedPtrField<Type>> : public list_caster<pb::RepeatedPtrField<Type>, Type> {};
}} // namespace pybind11::detail

Now that pybind11 knows how to bind RepeatedField and RepeatedPtrField as lists we can bind these as read-only properties of the respective classes in the module definition. Unfortunately some awkward casts are required here to help the compiler figure out the correct function overload to use.

namespace py = pybind11;

PYBIND11_MODULE(mymodule, m) {
    /* ... SNIP ... */
	py::class_<Value>(m, "Value")
		.def_property_readonly("data", (const pb::RepeatedField<double>&(Value::*)() const) &Value::data)
	;
	py::class_<ValuesResponse>(m, "ValuesResponse")
		.def_property_readonly("values", (const pb::RepeatedPtrField<Value>&(ValuesResponse::*)() const) &ValuesResponse::values)
	;
    /* ... SNIP ... */
}

Now some simple Python code to exercise the module:

import mymodule

# first, a native method (not shown) is invoked that returns an initialised ValuesResponse
response = get_values()

# a debugger watch shows that response.values is fully populated with valid data at this point in the flow of execution

for v in response.values:
    # now the fields of v are empty, and the entire list of response.values is corrupt
    for d in v.data:
        pass # when stepping over using the debugger, there is nothing in v.data any longer

Everything works as expected at first, the call to get_values() returns an instance of an object that appears to have a list of instances of Value and all fields appear to be initialised, but it seems like the bound objects get corrupted / all fields reset to 0 on subsequent reads of the bound values property (note: even len(response.values) will reset the list).

If I had to make a stab at a logical explanation, I would guess this is an issue of object lifetime, and perhaps Python reference couting is to blame. That's as far as I've got, but if anyone can shed further light on what's happening, I am very much open to ideas and inspiration.

Please let me know if a minimal working / broken example is required to reproduce the issue more easily.

[ljluestc]

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <type_traits>

namespace py = pybind11;

enum class ScopedCharEnum {
    A = 65,
    B = 66
};

PYBIND11_MODULE(test_enum, m) {
    py::enum_<ScopedCharEnum>(m, "ScopedCharEnum")
        .value("A", ScopedCharEnum::A)
        .value("B", ScopedCharEnum::B);
}

// In the template code related to the static assertion:
static_assert(
    std::is_same<py::enum_<ScopedCharEnum>::Scalar, signed char>::value || 
    std::is_same<py::enum_<ScopedCharEnum>::Scalar, unsigned char>::value ||
    std::is_same<py::enum_<ScopedCharEnum>::Scalar, char>::value, // Allow char type as valid
    "char should be cast to either signed char or unsigned char"
);