Making a C++ function pickleable?

[r-owen]

Issue description

Is there a practical way to make a C++ function pickleable? I want to do this in order to support pickle on my own classes with __reduce__.

In more detail, I am wrapping a hierarchy of C++ objects. I have a C++ factory function that will return a shared_ptr to a suitable object in the hierarchy given its serialization (the internal details don't matter to this question):

std::shared_ptr<Object> makeObject(std::string const &state);

As such it is trivial to use __reduce__ to add pickling support, and I think __getstate__ and __setstate__ would be a lot messier. I would like to add this Object.__reduce__:

    cls.def("__reduce__", [](Object const &self) {
        auto unpickleArgs = py::make_tuple( self.serialize(false));
        return py::make_tuple(py::cpp_function(makeObject), unpickleArgs);
    });

This compiles, and Object.__reduce__() runs correctly, but when I try to pickle objects I get this error: TypeError: can't pickle PyCapsule objects

I have worked around the problem by creating a functor class that does the same thing as makeObject and adding a __reduce__ method to that functor so it can be pickled:

class ObjectMaker {
public:
    ObjectMaker() = default;
    std::shared_ptr<Object> operator()(std::string const &state);
};
...
    py::class_<ObjectMaker, std::shared_ptr<ObjectMaker>> makerCls(mod, "ObjectMaker");
    makerCls.def(py::init<>());
    makerCls.def("__call__", &ObjectMaker::operator());
    makerCls.def("__reduce__", [makerCls](ObjectMaker const &self) {
        return py::make_tuple(makerCls, py::tuple());
    });

However, if somebody knows a simple way to add support to my makeObject function I could avoid the functor and its messy wrapper.

I have included a simple example below. There is no class hierarchy and so no need for __reduce__, but it shows the issue.

Reproducible example code

C++ Code

#include <pybind11/pybind11.h>

namespace py = pybind11;

class Temp {
public:
    std::string id() { return "a Temp"; }
};

Temp makeTemp() { return Temp(); }

PYBIND11_PLUGIN(temp) {
    py::module mod("temp");

    py::class_<Temp, std::shared_ptr<Temp>> cls(mod, "Temp");
    cls.def(py::init<>());
    cls.def("id", &Temp::id);
    cls.def("__reduce__", [](Temp const &self) {
                return py::make_tuple(py::cpp_function(makeTemp), py::make_tuple());
    });
    return mod.ptr();
}

Python code

import pickle
from example import temp

t = Temp()
print(t.id())
print(t.__reduce__())
print(pickle.dumps(t))

Results

This prints:

>>> example.Test().__reduce__()
(<built-in method  of PyCapsule object at 0x104381b10>, ())
>>> pickle.dumps(example.Test())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't pickle PyCapsule objects

[GoingMyWay]

Are there any methods to automatically set all the functions and classes pickeable?

[arthurp]

I'm working on a distributed database which has mobile Python code. The inability to pickle pybind11 functions forces all native function imports into the function that is being transferred, which is a pain and kinda messy. I'm gonna look at work arounds, but a pybind11 level fix would be great.

[urumican]

I would also like to try to use py::pickle to support serialization for an object with a python callback of type std::function in c++ code. But it seems when it calling getstate, the error still throw up with "PyCapsule cannot be pickled."

Here is how I reproduce this issue

class Person {
public:
    Person() {}
    Person(string name, int age, std::function<void(int, string)> f): name(name), age(age), f(f) {}

    void doSomething() {
        this->f(this->age, this->name);
    }
    
    string  getName() {
        return this->name;
    }

    int getAge() {
        return this->age;
    }

    string name;
    int age;
    std::function<void(int, string)> f;
};

PYBIND11_MODULE(human, m)
{
    py::class_<Person>(m, "Person")
        .def(py::init<string, int, std::function<void(int, string)>>())
        .def(py::init<>())
        .def_readwrite("name", &Person::name)
        .def_readwrite("age", &Person::age)
        .def("getName", &Person::getName)
        .def("getAge", &Person::getAge)
        .def("doSomething", &Person::doSomething)
        .def(py::pickle(
            [](Person &p) {
                std::cout << "creating serilizing." << std::endl;
                auto t = py::make_tuple(p.getName(), p.getAge(), p.f);
                std::cout << "done creating serilizing." << std::endl;
                return t;
            },
            [](py::tuple t) {
                if (t.size() != 3) {
                    throw runtime_error("invalid state!");
                }

                Person p(t[0].cast<string>(), t[1].cast<int>(), t[2].cast<std::function<void(int, string)>>());
                return p;
            }
        ));
}

[ShirAmir]

Is there any news regarding making the pybind11 functions pickleable?

[rwgk]

FWIW, I found one way to work around this issue, with pybind11 as is:

https://github.com/google/pywrapcc/pull/30099/commits/04e16a2d249f5d96a22dc9948c408810b54eb345

(I'm still exploring other options.)