snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

RDB Loader: raise an alert in case of degraded Redshift

Open chuwy opened this issue 4 years ago • 0 comments

This is a follow-up for #608. In addition to just checking if loading is progressed via stages, we also should be able to leverage STV_LOAD_STATE docs claim it's available to all users. So, if we see that stages are progressing very slowly - we can check the table to make sure that progress is slow. And if it is - raise an alarm.

Additionally there should be a isDegraded flag in global state, turned into true when we discover a slow DB. Whenever it's true we should at least stop folders monitoring to not degrade the warehouse completely.

chuwy avatar Nov 11 '21 13:11 chuwy