"cannot execute LISTEN during recovery" on Heroku

[ianterrell]

I'm using GoodJob on Heroku but have noticed this exception being raised occasionally. Google shows most hits around replication and swapping to a standby server; perhaps maintenance periods at Heroku, although I haven't set a specific window and can't see when it occurred to correlate with our errors.

It's raised during the LISTEN setup, obviously:

https://github.com/bensheldon/good_job/blob/284d67193772d336e5832304c93ab3f62fad9818/lib/good_job/notifier.rb#L205-L208

Is this something that GoodJob should be aware of and handle? Thanks!

[bensheldon]

Thanks for reporting! What is the error class that gets raised with that message? I can add that to the retryable list that will give it 30 seconds to succeed before it is sent to the error reporter.

[ianterrell]

@bensheldon It looks like it is ActiveRecord::StatementInvalid wrapping PG::ReadOnlySqlTransaction:

ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute LISTEN during recovery

Here's the backtrace: backtrace.log

& thanks!

[rsl]

this just happened for me as well. was there any further findings or solutions on this?

[bensheldon]

This exception is already part of GoodJob's LISTEN/NOTIFY reconnect list:

https://github.com/bensheldon/good_job/blob/90ea269a27c21f0425cce229c8d2519e9e2a6ff2/lib/good_job/notifier.rb#L25-L36

GoodJob will always attempt to reconnect perpetually on any exception, the only difference here is that these specific reconnect exceptions result in a longer reconnect interval and won't report the exception to the error handler until it reaches the threshold (5 seconds X 6 times = 1 minute), even after reported, it will continue to attempt to reconnect.