snowplow-rdb-loader
snowplow-rdb-loader copied to clipboard
Loader: add commands for pausing SQS pulling
If Redshift is faulty state it would make sense to stop receiving messages. Otherwise we need to resend many messages that were acked, but failed being loaded.
Is your idea to make the loader able to deal with signals to never stop it ?
Yes, but after writing this I realized that it's impossible to resume it with a message because our queue is FIFO, so if you sent a stop message, then few others shredding_complete then resume - the loader will never get the resume because it has to process shredding_complete first.
So unfortunately the only thing we can add is to pause for X minutes, which is not that useful.
Ah ok, I was talking about Linux signals. Indeed that's not possible to use SQS for this.