Project Antalya Roadmap 2025 - Real-Time Data Lakes

[hodgesrm]

Here's the current Project Antalya roadmap for 2025. This year the principal focus is adapting ClickHouse to use Iceberg as shared object storage and adding separation of storage and compute. All features are open source--there are no hold-backs.

Please suggest additional features and ideas in the comments to this issue. We also welcome contributions.

Performance:

• [x] Parquet metadata cache
• #586
• #636
• [x] Parquet native reader, v1
• https://github.com/ClickHouse/ClickHouse/pull/62966
• https://github.com/ClickHouse/ClickHouse/pull/70807
• https://github.com/ClickHouse/ClickHouse/pull/71055
• https://github.com/ClickHouse/ClickHouse/pull/71383
• https://github.com/ClickHouse/ClickHouse/pull/72105
• [ ] Parquet native reader, v3 (upstream) https://github.com/ClickHouse/ClickHouse/pull/70611
• [x] ListObjectsV2 cache #743
• [x] Iceberg table pruning in cluster requests #770
• [x] Iceberg files metadata cache (upstream) https://github.com/ClickHouse/ClickHouse/pull/77156
• [x] Iceberg partition pruning (upstream) https://github.com/ClickHouse/ClickHouse/pull/72044
• [x] Iceberg min/max pruning (upstream) https://github.com/ClickHouse/ClickHouse/pull/78242
• [ ] RowGroup adaptive size

Swarms:

• [x] Auto-discovery of swarm cluster nodes #629
• [x] Consistent hashing for object distribution to improve cache locality #709
• [x] Distributed object storage table engines #615
• [x] Swarm query syntax #712
• [ ] Swarm reliability/re-tries
• [ ] Swarm for writes
• [ ] Swarm for merges/optimize

Catalogs:

• [x] Open source catalog for Kubernetes https://github.com/Altinity/ice
• [x] AWS S3 Table support
• [x] Unity catalog support https://github.com/ClickHouse/ClickHouse/pull/76988
• [x] Glue catalog support https://github.com/ClickHouse/ClickHouse/pull/77257
• [ ] Cloudflare R2 Data Catalogs support
• [ ] Public datasets in Iceberg
• [ ] Use IAM roles to access s3 table function #688

Iceberg Writes:

• [x] Toolkit for loading files into Iceberg https://github.com/Altinity/ice
• [x] Support partitioning
• [x] Support ordering (see https://www.tabular.io/apache-iceberg-cookbook/data-engineering-table-write-order/)
• [ ] CREATE TABLE for Iceberg/DataLakeCatalog database engine
• [ ] INSERT INTO Iceberg table
• [ ] Use MergeTree buffer for frequent inserts into Iceberg (like async inserts but with much bigger buffer on disk)

Tiered Storage:

• [x] Wildcard support for object storage #789
• [x] Add support for hive partition style reads and writes https://github.com/ClickHouse/ClickHouse/pull/76802
• [x] Write MergeTree parts to Parquet #1009
• [ ] Write MergeTree partitions to Parquet
• [ ] Infinity table engine #990
• [ ] TTL to other table
• [ ] Merge tables with watermark
• [ ] Backup/restore for tiered tables (extension to Altinity Backup for ClickHouse aka clickhouse-backup)