oso
oso copied to clipboard
Enable bq2clickhouse as incremental
bq2cloudsql needs to be incremental in order to quickly update the cloudsql db.
For any of the "time" based fields we should be able to easily use partition data to handle incremental data synchronizations.
This won't cover things where we need to backfill. To cover for that, we should still be running a full push on a weekly basis (or whenever schema's change).
Saves time but not money
We discussed this, we are currently assuming that network and storage costs are negligible. In this case, the only thing we are saving is time, which Clickhouse imports are pretty fast at.
Let's put a pin in on this until it becomes an issue.