snowplow-rdb-loader icon indicating copy to clipboard operation
snowplow-rdb-loader copied to clipboard

Loader: add commands for pausing SQS pulling

Open chuwy opened this issue 4 years ago • 3 comments

If Redshift is faulty state it would make sense to stop receiving messages. Otherwise we need to resend many messages that were acked, but failed being loaded.

chuwy avatar May 05 '21 17:05 chuwy

Is your idea to make the loader able to deal with signals to never stop it ?

benjben avatar May 05 '21 19:05 benjben

Yes, but after writing this I realized that it's impossible to resume it with a message because our queue is FIFO, so if you sent a stop message, then few others shredding_complete then resume - the loader will never get the resume because it has to process shredding_complete first.

So unfortunately the only thing we can add is to pause for X minutes, which is not that useful.

chuwy avatar May 05 '21 19:05 chuwy

Ah ok, I was talking about Linux signals. Indeed that's not possible to use SQS for this.

benjben avatar May 05 '21 19:05 benjben