resque-scheduler icon indicating copy to clipboard operation
resque-scheduler copied to clipboard

Use with Redis 5.0 gem causes error loading schedule

Open ozzyaaron opened this issue 3 years ago • 6 comments

After upgrading from Redis gem v4.8.0 to v5.0.2 we got the following error on Heroku.

Our Redis URLs take the form redis://h:[email protected]:12345

Resque.schedule = YAML.load_file(Rails.root.join("config", "resque_schedule.yml"))

RedisClient::CommandError: ERR wrong number of arguments for 'auth' command

Reverting to Redis v4.8.0 corrects the issue.

ozzyaaron avatar Sep 06 '22 20:09 ozzyaaron

Seeing some issues with redis 5.x.x gem and BACKGROUND=yes too, scheduler exits silently with 0 code, but background process is not running, from what I gathered in strace -f child process fails with something like:

"NoMethodError: undefined method `reconnect' for #<Redis::Client redis://localhost:6379/0>

xeron avatar Sep 11 '22 08:09 xeron

I tried the latest version and it still silently fails in background mode (I wish logging wasn't so broken for resque-scheduler). strace shows some errors like:

rake aborted!

Redis::CannotConnectError: getaddrinfo_a: System error

...

Caused by:

RedisClient::CannotConnectError: getaddrinfo_a: System error

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/ruby_connection.rb:49:in `initialize'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `new'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `block in connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/middlewares.rb:8:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:652:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `block in connect'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `block in connect_with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `block in with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/tracer.rb:352:in `capture_segment_error'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `connect_with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:212:in `call_v'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/client.rb:73:in `call_v'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:167:in `block in send_command'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `synchronize'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `send_command'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/commands/connection.rb:21:in `ping'
/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
/.../shared/bundle/ruby/2.7.0/gems/resque-2.4.0/lib/resque/data_store.rb:83:in `reconnect'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:42:in `setup_backgrounding'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:17:in `setup'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/cli.rb:113:in `setup_env'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/tasks.rb:17:in `block (2 levels) in <main>'
/.../shared/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

Tasks: TOP => resque:scheduler

(See full trace by running task with --trace)

xeron avatar Oct 08 '22 19:10 xeron

After upgrading from Redis gem v4.8.0 to v5.0.2 we got the following error on Heroku.

Our Redis URLs take the form redis://h:[email protected]:12345


Resque.schedule = YAML.load_file(Rails.root.join("config", "resque_schedule.yml"))



RedisClient::CommandError: ERR wrong number of arguments for 'auth' command

Reverting to Redis v4.8.0 corrects the issue.

I think Redis 5.0 changed the way you specify the server and port. I'm not near a computer though so that may be something you want to look into.

PatrickTulskie avatar Oct 08 '22 19:10 PatrickTulskie

I tried the latest version and it still silently fails in background mode (I wish logging wasn't so broken for resque-scheduler). strace shows some errors like:


rake aborted!



Redis::CannotConnectError: getaddrinfo_a: System error



...



Caused by:



RedisClient::CannotConnectError: getaddrinfo_a: System error



/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/ruby_connection.rb:49:in `initialize'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `new'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `block in connect'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/middlewares.rb:8:in `connect'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:652:in `connect'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `block in connect'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `block in connect_with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `block in with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/tracer.rb:352:in `capture_segment_error'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `connect_with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `connect'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:212:in `call_v'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/client.rb:73:in `call_v'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:167:in `block in send_command'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `synchronize'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `send_command'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/commands/connection.rb:21:in `ping'

/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'

/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'

/.../shared/bundle/ruby/2.7.0/gems/resque-2.4.0/lib/resque/data_store.rb:83:in `reconnect'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:42:in `setup_backgrounding'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:17:in `setup'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/cli.rb:113:in `setup_env'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/tasks.rb:17:in `block (2 levels) in <main>'

/.../shared/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'



Tasks: TOP => resque:scheduler



(See full trace by running task with --trace)

This one is a new one to me. I wish I knew what that system error was. Reverting to redis 4.8 fixes it?

PatrickTulskie avatar Oct 08 '22 19:10 PatrickTulskie

This one is a new one to me. I wish I knew what that system error was. Reverting to redis 4.8 fixes it?

Yes.

xeron avatar Oct 10 '22 18:10 xeron

FYI I fixed this by changing redis connection string from:

Resque.redis = 'localhost:6379'

to

Resque.redis = '127.0.0.1:6379'

Not sure what's the deal with getaddrinfo_a: System error. It only happens when started by capistrano-redis. I couldn't reproduce it manually.

xeron avatar Jun 16 '23 06:06 xeron