Add support for custom debug adapters from C++/Python API

[Copilot]

trafficstars

This PR implements comprehensive support for creating custom debug adapters from the C++ and Python API side, allowing developers to extend the Binary Ninja Debugger with new protocols, targets, and specialized debugging scenarios.

Architecture

The implementation uses a three-layer bridge architecture:

1. FFI Layer: C-style callback structures for maximum compatibility across languages
2. Core Bridge: CustomDebugAdapter and CustomDebugAdapterType classes that forward calls to user-provided callbacks
3. API Layer: Object-oriented C++ and Python base classes providing clean interfaces

Key Features

• Comprehensive FFI Interface: 48 debug adapter callbacks covering all functionality (execution, debugging, memory access, register manipulation, etc.) plus 5 adapter type callbacks for registration and capability checking
• C++ API: Type-safe base classes CustomDebugAdapter and CustomDebugAdapterType with automatic type conversions
• Python API: Complete Python bindings with proper error handling and Pythonic interfaces
• Registration System: Functions to register custom adapters with the debugger core

Usage Examples

C++ Implementation:

class MyAdapter : public BinaryNinjaDebuggerAPI::CustomDebugAdapter {
public:
    bool Execute(const std::string& path) override {
        // Custom implementation
        return true;
    }
    // ... implement other required methods
};

class MyAdapterType : public BinaryNinjaDebuggerAPI::CustomDebugAdapterType {
public:
    MyAdapterType() : CustomDebugAdapterType("MyCustomAdapter") {}
    
    std::unique_ptr<CustomDebugAdapter> Create(Ref<BinaryView> data) override {
        return std::make_unique<MyAdapter>();
    }
    
    bool CanConnect(Ref<BinaryView> data) override { return true; }
    bool CanExecute(Ref<BinaryView> data) override { return false; }
};

// Registration
static MyAdapterType adapterType;
adapterType.Register();

Python Implementation:

from debugger.customdebugadapter import CustomDebugAdapter, CustomDebugAdapterType

class MyPythonAdapter(CustomDebugAdapter):
    def execute(self, path: str) -> bool:
        # Custom implementation
        return True
    
    def attach(self, pid: int) -> bool:
        # Custom implementation
        return True
    
    # ... implement other required methods

class MyPythonAdapterType(CustomDebugAdapterType):
    def __init__(self):
        super().__init__("MyPythonAdapter")
    
    def create(self, bv):
        return MyPythonAdapter()
    
    def can_connect(self, bv):
        return True

# Registration
adapter_type = MyPythonAdapterType()
adapter_type.register()

Implementation Details

• Bridge Pattern: Core CustomDebugAdapter class forwards all virtual method calls to user-provided FFI callbacks
• Memory Management: Proper lifetime management with reference counting and cleanup callbacks
• Error Handling: Comprehensive error handling with fallback behaviors for optional methods
• Type Safety: Full type conversions between C++ types and FFI structures
• Documentation: Complete API documentation with examples in docs/custom_debug_adapters.md

Testing

All implementation tests pass (6/6):

• FFI interface validation
• Core bridge implementation verification
• C++ API functionality testing
• Python bindings validation
• Example code compilation
• Integration testing

This enables extending the Binary Ninja Debugger with custom protocols (e.g., proprietary debug interfaces), specialized targets (e.g., embedded systems, virtual machines), and unique debugging scenarios (e.g., trace analysis, simulation environments).

Fixes #370.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}