dbt-starrocks icon indicating copy to clipboard operation
dbt-starrocks copied to clipboard

Add delete+insert incremental materialization strategy

Open chris-celerdata opened this issue 2 months ago • 0 comments

Description

Currently, dbt-starrocks supports ['append', 'insert_overwrite', 'dynamic_overwrite', 'microbatch']. Another extremely common strategy is delete+insert. This strategy modifies rows by first deleting records that match specified conditions, then inserting the new/updated records. Support for row-level deletion does exist.

Proposed Implementation

Native StarRocks Tables

The delete+insert strategy can be implemented immediately for native StarRocks tables using the existing DELETE statement syntax:

DELETE FROM target_table WHERE <unique_key_condition>;
INSERT INTO target_table SELECT * FROM temp_table;

External Tables

Implementation for external tables requires upstream changes in StarRocks core to support row-level modifications. For example, in Iceberg V2, delete operations are supported through position deletes and equality deletes, but StarRocks would need to expose this functionality.

Acceptance Criteria

  • [ ] Add delete+insert to supported incremental strategies for native StarRocks tables
  • [ ] Implement logic to generate appropriate DELETE and INSERT statements based on unique_key configuration
  • [ ] Add tests covering edge cases (null keys, multiple unique keys, etc.)

Additional Notes

Another strategy is merge, but that would require MERGE INTO or UPDATE to be supported, which does not look like a planned feature in StarRocks core. Therefore, this issue will only focus on delete+insert.

chris-celerdata avatar Oct 16 '25 20:10 chris-celerdata