new feature: provides a way to control memory allocation and deallocation

[tbEgg]

Feature Description

OpenDAL currently does not support allowing callers to take control of memory management, which is crucial for programs based on C and C++.

Problem and Solution

Enable users to define a custom memory allocator that can replace the default Rust memory allocation mechanisms.

Additional Context

No response

Are you willing to contribute to the development of this feature?

• [X] Yes, I am willing to contribute to the development of this feature.

[Xuanwo]

cc @PragmaTwice @xyjixyjixyji @silver-ymz, is it possible to pass an Allocator from the C side? I haven't seen such an attempt before; perhaps we are the pioneers in this area. It should be exciting.

[xyjixyjixyji]

I think at this point there is only one solution if we want to use a custom allocator.

We expose an API like this

type AllocFn = unsafe extern "C" fn(size: usize) -> *mut u8;
type DeallocFn = unsafe extern "C" fn(ptr: *mut u8, size: usize);

#[no_mangle]
pub extern "C" fn opendal_set_global_allocator(alloc: AllocFn, dealloc: DeallocFn);

And users call opendal_set_global_allocator() before any dynamic memory allocation, we can also force this so that this can only be called once.

If everyone agree on this I could do this in a few days. \cc @Xuanwo @tbEgg

[silver-ymz]

I have 2 questions.

The GlobalAlloc trait in standard library is defined as follows:

pub unsafe trait GlobalAlloc {
    // Required methods
    unsafe fn alloc(&self, layout: Layout) -> *mut u8;
    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout);

    // Provided methods
    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8;
    unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8;
}

1. Should alignment information be exposed to C functions?

Should we expose the alignment parameter to C functions in the following way?

type AllocFn = unsafe extern "C" fn(size: usize, alignment: usize) -> *mut u8;
type DeallocFn = unsafe extern "C" fn(ptr: *mut u8, size: usize, alignment: usize);

For example, the GNU malloc function aligns to 8 bytes on 32-bit systems and 16 bytes on 64-bit systems. However, Rust allows more explicit alignment specifications such as #[repr(align(32))]. If we do not expose the alignment parameter, it seems that we would need to perform additional work to handle alignment internally.

2. Should alloc_zeroed and realloc be exposed for further optimization?

Modern allocators often have specialized functions to optimize memory management. For example, mimalloc provides mi_zalloc and mi_realloc. Should we expose these functions as optional arguments, allowing them to be nullptr if not implemented?

[Xuanwo]

If everyone agree on this I could do this in a few days.

Hi, I suggest that we don't rush this. I want to understand how users will integrate this feature into their own app.