octopus
octopus copied to clipboard
Delayed Job + Octopus 0.10.2 issues on Rails 5.2 (undefined method `any?' for nil:NilClass)
It looks like Delayed Job and Octopus are not happy with each other out of the box. I know both are fairly common gems used by the Rails community so I thought I'd go ahead and log an issue here with a collection of workarounds cobbled together from various other documented issues.
In particular, it looks like forking in general is a problem and documented here: https://github.com/thiagopradi/octopus/issues/489 and here: https://github.com/thiagopradi/octopus/issues/511 but these are documented as Puma issues, when in fact it's true for delayed_job as well.
In addition, there is also a shard selection issue documented here: https://github.com/tchandy/octopus/issues/241
In case anyone wanders into this Issue, here's initializer that I've put together that seems to fix things:
# from this: https://github.com/tchandy/octopus/issues/241
# this prevents delayed job from pulling from the wrong shard
module Delayed
module Backend
module ActiveRecord
class Job < ::ActiveRecord::Base
class << self
alias_method :reserve_without_octopus, :reserve
def reserve(worker, max_run_time = Worker.max_run_time)
Octopus.using(:master) do
reserve_without_octopus(worker, max_run_time)
end
end
end
end
end
end
end
# from this: https://github.com/thiagopradi/octopus/issues/489
# and also this: https://github.com/thiagopradi/octopus/issues/511
# Forking in DJ seems to cause issues with the connections
module Octopus
class Proxy
alias_method :safe_connection_without_fork_check, :safe_connection
def safe_connection_with_fork_check(connection_pool)
retries ||= 0
safe_connection_without_fork_check(connection_pool)
rescue NoMethodError => e
ActiveRecord::Base.establish_connection
ActiveRecord::Base.connection.initialize_shards(Octopus.config)
retry if (retries += 1) < 2
end
alias_method :safe_connection, :safe_connection_with_fork_check
def connected?
retries ||= 0
shards.any? { |_k, v| v.connected? }
rescue NoMethodError
proxy_config.reinitialize_shards
retry if (retries += 1) < 2
end
end
end
Upgrading a Rails 5.1 system to 5.2, and encountered this issue. Switching to @mikwat's fork seems to solve it.
i was having an issue with unicorn
which also uses forking. a much simpler solution in that case is to use its after_fork
hook to re-establish octopus connections:
# config/unicorn.rb
after_fork do |server, worker|
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
ActiveRecord::Base.connection.initialize_shards(Octopus.config) if Octopus.enabled?
end
end