apache
Is there any actual conversion implementation for arrow and parquet?
We found that there are only schema conversions under the parqeut-arrow, so we wanted to ask if there is any code that includes the actual data conversions between parquet and arrow.
If possible to use C++, I think parquet-cpp in the Apache Arrow is the best solution to your case: https://arrow.apache.org/docs/cpp/parquet.html
Yes, We know there is a c++ implementation here, but I was wondering if there is a corresponding implementation for java, since all our code is java .
parquet-arrow The library looks like it's meant to do this, But I don't know why it's always just the schema part.
I think conversion between parquet and arrow is a valid use case. The parquet-java provides built-in row-level interfaces like avro/thrift/protobuf. Other parquet (Java) implementations (Presto/Trino/Spark) simply leverage the page & metadata reader/writer from this library to build extensions. Extending native arrow support would be a welcome extension to this library, IMO.
So, the library parquet-arrow hasn't been used yet? because it only has schema mappings.
And We've looked at transformations in Spark, which are missing some types due to limitations in Spark, such as uint.So it's not really a universal conversion.
It seems that iceberg has an arrow implementation.
Yes, but it does not support reading repetition levels and v2 encodings.
I've ported Iceberg's implementation to parquet-arrow and removed the concepts specific to Iceberg. If someone is interested in this, they are welcome to make contributions.