databend icon indicating copy to clipboard operation
databend copied to clipboard
verified publisher icon datafuselabs

Feature: Allow COPY FROM AVRO file

Open Xuanwo opened this issue 3 years ago • 7 comments

Summary

https://avro.apache.org/

Apache Avro™ is the leading serialization format for record data, and first choice for streaming data pipelines. It offers excellent schema evolution, and has implementations for the JVM (Java, Kotlin, Scala, …), Python, C/C++/C#, PHP, Ruby, Rust, JavaScript, and even Perl.

Xuanwo avatar Oct 02 '22 18:10 Xuanwo

Figuered out how to implement, working on it.

ClSlaid avatar Oct 19 '22 16:10 ClSlaid

Is there any word on whether this will be available in an upcoming release?

inviscid avatar Jul 10 '24 18:07 inviscid

Is there any word on whether this will be available in an upcoming release?

Hi, there's no effort for this task. Perhaps you could contact @BohuTANG for your needs?

Xuanwo avatar Jul 12 '24 06:07 Xuanwo

@inviscid We haven't made any plans regarding this yet. Could you provide more details about these requirements?

BohuTANG avatar Jul 12 '24 06:07 BohuTANG

We have many files in avro format and would like to be able to create a stage with file_format = (type = AVRO)

rdonnelly3 avatar Aug 21 '24 13:08 rdonnelly3

There is a Rust crate for handling Avro files. Hopefully that could be used and extend the current import/stage with an Avro reader. https://crates.io/crates/apache-avro

inviscid avatar Dec 05 '24 13:12 inviscid

cc @sundy-li

ZhiHanZ avatar Feb 18 '25 15:02 ZhiHanZ