arrow icon indicating copy to clipboard operation
arrow copied to clipboard

[Java] VectorSchemaRoot convertion into ArrowArrayStream

Open igor-suhorukov opened this issue 2 years ago • 6 comments

Describe the enhancement requested

DuckDB support ArrowArrayStream as virtual table. JDBC driver of DuckDB has "register" ArrowStream method on DuckDBConnection.

VectorSchemaRoot convertion into ArrowArrayStream object will allows perform complex SQL transformation with CTE,window function on existing Arrow data from java process without data format transformation and also as workaround to write parquet file from VectorSchemaRoot.

try (DuckDBConnection connection = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:")){
    connection.registerArrowStream("arrow_stream", arrowArrayStream);
    try (Statement preparedStatement = connection.createStatement()){
        preparedStatement.executeUpdate("COPY (SELECT * FROM arrow_stream) TO '"+filePath+"' (FORMAT 'PARQUET', CODEC 'ZSTD')");
    }
}

@lidavidm do you have idea how to implement such conversion?

Component(s)

Java

igor-suhorukov avatar Mar 23 '23 05:03 igor-suhorukov