arthur-redshift-etl icon indicating copy to clipboard operation
arthur-redshift-etl copied to clipboard

Old data is (usually) better than no data for sources

Open bhtucker opened this issue 7 years ago • 0 comments

In a concurrent load, extracts may be observed to fail or may stall for long enough that the load process 'times out' waiting for them.

When this happens, the last-loaded version of the data source may be available elsewhere in the warehouse (ie, if loading in staging, a copy is sitting in the non-prefixed 'standard' position). Rather than fail the relation, we should try to load our new table using the data from that existing table.

bhtucker avatar Dec 20 '17 18:12 bhtucker