oso icon indicating copy to clipboard operation
oso copied to clipboard

Enable bq2clickhouse as incremental

Open ravenac95 opened this issue 1 year ago • 2 comments

bq2cloudsql needs to be incremental in order to quickly update the cloudsql db.

For any of the "time" based fields we should be able to easily use partition data to handle incremental data synchronizations.

This won't cover things where we need to backfill. To cover for that, we should still be running a full push on a weekly basis (or whenever schema's change).

ravenac95 avatar Feb 14 '24 17:02 ravenac95

Saves time but not money

ryscheng avatar Feb 20 '24 17:02 ryscheng

We discussed this, we are currently assuming that network and storage costs are negligible. In this case, the only thing we are saving is time, which Clickhouse imports are pretty fast at.

Let's put a pin in on this until it becomes an issue.

ryscheng avatar Aug 21 '24 01:08 ryscheng