dbt-athena icon indicating copy to clipboard operation
dbt-athena copied to clipboard

Zero-downtime for table recalculation within dbt run

Open SOVALINUX opened this issue 2 years ago • 1 comments

Hi I'm sorry if I'm missing something - I'm new in dbt and just trying to switch the project from custom data model management to dbt

Does dbt-athena supports table recalculation without downtime while it's being recalculated?

What workaround I used to do in my own script: to give every CTAS an unique name (i.e. _) and have a helper view with a fixed name () and simple query select * from <name>_<random> - so once new version of ctas is calculated I switch a view to the new version and drop the old one. All usages reference only view and the whole thing works without downtime

Will appreciate your thoughts and comments!

SOVALINUX avatar Apr 10 '22 21:04 SOVALINUX

I had a shot at implementing a PoC for this feature as well, maybe that could help! My code is at https://github.com/iasoon/dbt-athena/commit/cca66da328058e40ec5490cdb7a5a4589b042440. It seems to work in simple tests, but it's not battle-tested :)

iasoon avatar May 11 '22 09:05 iasoon