spiraldb
Add support to Iceberg for Vortex files
We recently achieved a v0 integration of Vortex for Iceberg Java that consists of
- a Java SDK, contained in this repo
- An Iceberg Java fork so we can collaborate with the community on pluggable file formats in Iceberg, based on https://github.com/pvary/iceberg/tree/file_Format_api_without_base
So far the results are promising, with up to 40% reduction in overall TPC-H SF=100 runtime and 5x improvement for some queries.
Vortex's performance stats and innovative tech are pretty impressive, and I'm planning to give it a spin in our production environment. My main concern is that when Vortex integrates with Iceberg, it uses a forked branch of Iceberg, which can't be merged back upstream, right? Just wondering if we've got any plans for that. Thanks a bunch!
The plan is for Iceberg to support other file formats https://github.com/apache/iceberg/issues/12225. Once that's done upstream our iceberg version can depend on stable iceberg
