delayed_job dies with database connection issues

Open jconley opened this issue 14 years ago • 15 comments

Received this in our delayed_job.log... and then the DJ process dies. My best guess so far is there was a momentary network disruption between firewalls of our app and db networks.

<ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has gone away: UPDATE `delayed_jobs` SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:12345') >

Would be nice if DJ had more robust connection handling... i.e. catch this issue, and retry the connection until it is successful... otherwise we have to write a script to restart DJ periodically, or detect when jobs in the queue are stale.

Sep 27 '09 19:09 jconley

Experiencing the same issue even when the db is on the same box.

Mar 25 '10 00:03 smasry

Try setting 'reconnect: true' in your database.yml for the appropriate environment; under Rails 2.3.x that should alleviate the problem. (I'm setting that up now myself, so I'll know for sure in a few days, but it makes sense.)

Mar 26 '10 05:03 cyberfox

"reconnect: true" works, but leads to problems with transactions, so I don't think that is a proper solution

Jul 07 '10 09:07 kelyar

True, my use case was specific: the only database actions being done in the application were against the DJ table via DJ, so the issue with transactions was not relevant.

You could still do it if your DJ database was on a separate connection from your normal DB, but that certainly becomes more complex to set up.

Jul 09 '10 08:07 cyberfox

Same issue with rails 3 beta4 .... Can't make it work.

$ script/delayed_job start delayed_job: process with pid 20648 started.

=> SQL (111.9ms) UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:desk pid:20648') Mysql::Error: Lost connection to MySQL server during query: SELECT delayed_jobs.* FROM delayed_jobs WHERE ((run_at <= '2010-08-06 14:26:41' AND (locked_at IS NULL OR locked_at < '2010-08-06 10:26:41') OR locked_by = 'delayed_job host:desk pid:20648') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 5

And when I start DJ with rake, every things working:

$ rake jobs:work [Worker(host:desk pid:20663)] Starting job worker

=>SQL (24.5ms) SHOW TABLES DEPRECATION WARNING: RAILS_DEFAULT_LOGGER is deprecated! Use Rails.logger instead. (called from say at /home/www/.bundle/ruby/1.8/bundler/gems/delayed_job-0ca4e5f2bd8a5e17dca7c1abab692a049c1669e1-rails3-mailer/lib/delayed/worker.rb:161) 2010-08-06T16:28:04+0200: [Worker(host:desk pid:20663)] Starting job worker

Aug 06 '10 14:08 TeuF

I'm seeing the same behavior with Rails 2.3.5 and the collectiveidea fork of delayed_job (version 2.0.3). When starting from script/delayed_job, it immediately dies with a "MySQL server has gone away" message, but when starting using the work:jobs rake task, it starts up properly.

Rails application log logs this in conjunction with the failed startup:

Delayed::Backend::ActiveRecord::Job Update (0.0ms) Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:2293') Mysql::Error: MySQL server has gone away: UPDATE delayed_jobs SET locked_by = null, locked_at = null WHERE (locked_by = 'delayed_job host:myhost pid:2293')

Sep 26 '10 05:09 threetee

Solution that I found somewhere on internet was to downgrade daemons gem (which is internal dependancy of DJ, I believe) from 1.1.0 to 1.0.10. That completely solved problems with dying delayed job.

thank you, Anonymous!

Nov 15 '10 23:11 kelyar

I've also experienced many "Mysql Server gone away errors" with activerecord. The fix was to increase the 'max_allowed_packet'. If the stacktrace for an error was too large mysql would terminate the connection.

Apr 11 '11 19:04 smasry

suckage, this is still an issue. I've always hated the daemons gem