pybind11 icon indicating copy to clipboard operation
pybind11 copied to clipboard

[FEATURE REQUEST]: Dynamic class names (`pybind11::class_` constructor taking `std::string`)

Open burnpanck opened this issue 2 years ago • 2 comments

Required prerequisites

  • [X] Make sure you've read the documentation. Your issue may be addressed there.
  • [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
  • [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.4

Problem description

I often find myself wrapping C++ templates using pybind11. In this case, I can write a template function that instantiates and registers a specific form of the template. Of course, if I do this for multiple instantiations, I end up with multiple python classes, and those should have distinct names. I would like to generate these names in the template function at runtime. The problem is that the class_ constructors only take const char * for the name, providing no way of managing the dynamic memory where the runtime-generated name resides (a naive use of std::string::c_str() obviously leads to dangling references).

Python's type system probably wasn't designed with C++ templates in mind, so it likely doesn't provide an direct way of resource management for the name of a type object. However, we can make a workaround using weak references, following the example given in the pybind11 documentation. I have been using the following:

template <typename Cls, typename... Options, typename... Args>
auto make_class(py::module_ &m, std::string name, Args &&...args) {
  auto holder = std::make_unique<std::string>(std::move(name));
  auto ret = py::class_<Cls, Options...>(m, holder->c_str(), std::forward<Args>(args)...);
  py::cpp_function cleanup_callback([holder = std::move(holder)](py::handle weakref) mutable {
    holder.reset();
    weakref.dec_ref();
  });
  (void)pybind11::weakref(ret, cleanup_callback).release();
  return ret;
}

I wonder if something like this could be reasonably included in a dedicated constructor?

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression

burnpanck avatar Aug 03 '23 17:08 burnpanck


#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <memory>

namespace py = pybind11;

// Helper function to create a py::class_ with a dynamic class name
template <typename Cls, typename... Options, typename... Args>
auto make_class(py::module_ &m, std::string name, Args&&... args) {
    // Use a unique_ptr to manage the lifetime of the dynamically generated name
    auto holder = std::make_unique<std::string>(std::move(name));
    
    // Create the py::class_ object, using holder->c_str() for the name
    auto ret = py::class_<Cls, Options...>(m, holder->c_str(), std::forward<Args>(args)...);

    // Create a cleanup callback that will be invoked when the weakref is garbage collected
    py::cpp_function cleanup_callback([holder = std::move(holder)](py::handle weakref) mutable {
        holder.reset();  // Release the string memory
        weakref.dec_ref();  // Decrease the reference count for the weakref
    });

    // Create a weak reference for the class to ensure proper cleanup of the dynamically allocated string
    (void)pybind11::weakref(ret, cleanup_callback).release();

    return ret;
}

class MyClass {
public:
    MyClass(int x) : x(x) {}
    int x;
};

PYBIND11_MODULE(example, m) {
    // Use the make_class helper function to dynamically generate a class name at runtime
    make_class<MyClass>(m, "DynamicClassName", py::init<int>());
}
`#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <memory>

namespace py = pybind11;

// Helper function to create a py::class_ with a dynamic class name
template <typename Cls, typename... Options, typename... Args>
auto make_class(py::module_ &m, std::string name, Args&&... args) {
    // Use a unique_ptr to manage the lifetime of the dynamically generated name
    auto holder = std::make_unique<std::string>(std::move(name));
    
    // Create the py::class_ object, using holder->c_str() for the name
    auto ret = py::class_<Cls, Options...>(m, holder->c_str(), std::forward<Args>(args)...);

    // Create a cleanup callback that will be invoked when the weakref is garbage collected
    py::cpp_function cleanup_callback([holder = std::move(holder)](py::handle weakref) mutable {
        holder.reset();  // Release the string memory
        weakref.dec_ref();  // Decrease the reference count for the weakref
    });

    // Create a weak reference for the class to ensure proper cleanup of the dynamically allocated string
    (void)pybind11::weakref(ret, cleanup_callback).release();

    return ret;
}

class MyClass {
public:
    MyClass(int x) : x(x) {}
    int x;
};

PYBIND11_MODULE(example, m) {
    // Use the make_class helper function to dynamically generate a class name at runtime
    make_class<MyClass>(m, "DynamicClassName", py::init<int>());
}

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <memory>

namespace py = pybind11;

// Helper function to create a py::class_ with a dynamic class name
template <typename Cls, typename... Options, typename... Args>
auto make_class(py::module_ &m, std::string name, Args&&... args) {
    // Use a unique_ptr to manage the lifetime of the dynamically generated name
    auto holder = std::make_unique<std::string>(std::move(name));
    
    // Create the py::class_ object, using holder->c_str() for the name
    auto ret = py::class_<Cls, Options...>(m, holder->c_str(), std::forward<Args>(args)...);

    // Create a cleanup callback that will be invoked when the weakref is garbage collected
    py::cpp_function cleanup_callback([holder = std::move(holder)](py::handle weakref) mutable {
        holder.reset();  // Release the string memory
        weakref.dec_ref();  // Decrease the reference count for the weakref
    });

    // Create a weak reference for the class to ensure proper cleanup of the dynamically allocated string
    (void)pybind11::weakref(ret, cleanup_callback).release();

    return ret;
}

class MyClass {
public:
    MyClass(int x) : x(x) {}
    int x;
};

PYBIND11_MODULE(example, m) {
    // Use the make_class helper function to dynamically generate a class name at runtime
    make_class<MyClass>(m, "DynamicClassName", py::init<int>());
}

ljluestc avatar Nov 23 '24 22:11 ljluestc

@ljluestc: What exactly are you trying to say with the code-examples you provided? IMHO, it looks like you just copied the implementation I provided above, added a few comments, and made a full working example out of it (also, it looks like there is a tripple copy-paste in there). As it stands, that doesn't add anything to the discussion but only distracts from it with a large blob of code... I'd appreciate if you could correct the code example, and add a few words on why you post that code.

My intention was to discuss if it were reasonable to add a corresponding constructor to py::class_ instead, which would take a std::string, and basically did all of my make_class internally as needed.

burnpanck avatar Nov 26 '24 10:11 burnpanck