atum icon indicating copy to clipboard operation
atum copied to clipboard

Processing checkpoints and destructive operations on Dataframe

Open Zejnilovic opened this issue 3 years ago • 0 comments

We need to come up with a way to process checkpoints between destructive operations. So if I do filtering of the data and lose some rows, I can flag the next checkpoint as "post-destructive", "deduplication" or something similar.

This will pay off a lot when we implement validation #95 and have the ability to attach the control measures onto data frame instead of spark session.

Zejnilovic avatar Jul 23 '21 09:07 Zejnilovic