clickhouse.rs icon indicating copy to clipboard operation
clickhouse.rs copied to clipboard

Is there a way to directly fetch a batch rather than a row?

Open acking-you opened this issue 6 months ago • 1 comments

I've noticed that most examples return data in a row-by-row format. Is there a way to get it in batches instead?🤔 For example, something like this process could retrieve an arrow formatted batch:

use clickhouse::Client;
use tokio;
use arrow::ipc::reader::StreamReader;
use std::io::Cursor;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Creating a ClickHouse Client
    let client = Client::default()
        .with_url("http://localhost:8123")
        .with_database("default")
        .with_user("default")
        .with_password("");

    // Execute a query and get the data in Arrow IPC format
    let response = client.query("SELECT id, name, value FROM my_table FORMAT ArrowStream")
        .fetch_raw()
        .await?;

    // Convert response data to Arrow StreamReader
    let cursor = Cursor::new(response);
    let mut reader = StreamReader::try_new(cursor)?;

    // Processing Arrow batch data
    while let Some(batch) = reader.next() {
        let batch = batch?;
        println!("{:?}", batch);
    }

    println!("Arrow IPC read completed.");

    Ok(())
}

acking-you avatar Jul 30 '24 11:07 acking-you