c-questdb-client icon indicating copy to clipboard operation
c-questdb-client copied to clipboard

Support for ND Arrays in ILP

Open amunra opened this issue 9 months ago • 0 comments

Following https://github.com/questdb/c-questdb-client/issues/89, implement support for ND arrays in ILP.

Guidelines

Rust API

  • We should introduce a new trait for what constitutes an ND array.
  • Via feature flags, we should introduce optional support the ndarray crate.

Something like this:

use ndarray::array;
let arr = array![
    [1, 2, 3],
    [4, 5, 6]
];
buf.column_arr(arr)?;
pub trait NdArrayView<T>
where
    T: ArrayElement,
{
    /// Return the rank of the array
    fn ndim(&self) -> usize;

    /// Return the size of the `index` dimension
    fn dim(&self, index: usize) -> Option<usize>;
    
    /// Write the array data in row-major order to the provided buffer.
    /// The provided `buffer` will be pre-sized to the exact size as computed
    /// from the shape and element size.
    /// You may _not_ assume that the buffer's start is aligned to `T`.
    fn write_row_major_buf(&self, buff: &mut [u8]);
}

pub trait ArrayElement: Copy + 'static {}
impl ArrayElement for i32 {}
impl ArrayElement for i64 {}
impl ArrayElement for f32 {}
impl ArrayElement for f64 {}

pub fn column_arr<T, A>(arr: &A) -> Result<...>
where
    T: ArrayElement,
    A: NdArrayView<T>,
{
    ...
}

Then the impl:

#[cfg(feature = "ndarray")]
impl<T> NdArrayView<T> for ndarray::ArrayView<'_, T, '_>
where
    T: ArrayElement,
{
    fn ndim(&self) -> usize {
        self.ndim()
    }

    fn dim(&self, index: usize) -> Option<usize> {
        self.shape().get(index)
    }

    fn write_row_major_buf(&self, buf: &mut [u8]) {
        // figure out if already in standard layout row major:
        //   -> do a byte copy
        // if strided:
        //   -> iterate in row major order, copy value by value
    }
}

C and C++ APIs:

The other API will need an exploded set of APIs for different types. Each type (integer double etc) will take a buffer for the strides and a buffer for the data. E.g.:

LINESENDER_API
bool line_sender_buffer_column_f64_arr(
    line_sender_buffer* buffer,
    line_sender_column_name name,
    size_t rank,
    const size_t* shape,
    size_t data_buffer_len,
    const uint8_t* data_buffer,
    line_sender_error** err_out);

In C++ we can use some template specialisation to go back to an API more similar to Rust's. We should also use std::byte instead of uint8_t to refer to the buffer.

amunra avatar Mar 11 '25 11:03 amunra