starrocks-connector-for-apache-spark icon indicating copy to clipboard operation
starrocks-connector-for-apache-spark copied to clipboard

Published 20 hours ago • verified publisher icon StarRocks

[feature] support spark overwrite into starrocks

Open ZhendongBai opened this issue 1 year ago • 2 comments

What type of PR is this：

  • [ ] BugFix
  • [x] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Which issues of this PR fixes ：

Fixes #125

Problem Summary(Required) ：

  1. We implement the overwrite operation by using a temporary partition and atomically replacing the partition. After the job finishes, we create a new table and write to it.
  2. The swap table statement can be used to perform the overwrite operation on the entire table.
  3. However, expression partitioning does not allow the creation of partitions through the ALTER TABLE statement. This limitation means that tables using expression partitioning do not support the overwrite operation.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [x] This pr will affect users' behaviors
  • [x] This pr needs user documentation (for new or modified features or behaviors)
  • [ ] I have added documentation for my new feature or new function

ZhendongBai avatar Nov 14 '24 08:11 ZhendongBai

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Nov 14 '24 08:11 CLAassistant

@banmoy are you planning to introduce this capabillity?

pradomota avatar Apr 23 '25 07:04 pradomota