starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

Support iceberg v2 table

Open caneGuy opened this issue 2 years ago • 1 comments

Enhancement

v2 format can support update for iceberg table related with #1030

We will use c++ to implement the merge on read logic

Phase 1: Key points

  1. Only support Parquet and Orc
  2. Schedule data file scan range together with related delete files, process them together on BE
  3. Support position pushdown separately for Parquet and Orc
  4. Only support position delete file
  • [ ] Support get delete file information in Planner and send to BE
  • [ ] Parquet: Support virtual column _POS for position delete file
  • [ ] Parquet: Support position pushdown
  • [ ] Orc: Support virtual column _POS for position delete file
  • [ ] Orc: Support position pushdown
  • [ ] Implement merge on read logic

Phase 2x:

  • [ ] Support equality delete file
  • [ ] Add Avro reader

caneGuy avatar Feb 17 '22 06:02 caneGuy

cc @choury @DorianZheng @mxdzs0612 @stephen-shelby

caneGuy avatar Aug 10 '22 12:08 caneGuy

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!

github-actions[bot] avatar Feb 13 '23 11:02 github-actions[bot]