chatwoot icon indicating copy to clipboard operation
chatwoot copied to clipboard

Redis Sentinel not working

Open hazem-nasser96 opened this issue 10 months ago • 3 comments

Describe the bug

Latest version(4.2) can't connect to redis using sentinel. Tested in docker setup.

rails-container keeps restarting with error log:

rails-default-1 | + rm -rf /app/tmp/pids/server.pid rails-default-1 | + rm -rf '/app/tmp/cache/*' rails-default-1 | + echo 'Waiting for postgres to become ready....' rails-default-1 | Waiting for postgres to become ready.... rails-default-1 | + docker/entrypoints/helpers/pg_database_url.rb rails-default-1 | + export 'POSTGRES_PORT=5432' rails-default-1 | + PG_READY='pg_isready -h postgres-default -p 5432 -U postgres' rails-default-1 | + pg_isready -h postgres-default -p 5432 -U postgres rails-default-1 | + echo 'Database ready to accept connections.' rails-default-1 | + bundle install rails-default-1 | postgres-default:5432 - accepting connections rails-default-1 | Database ready to accept connections. rails-default-1 | Bundle complete! 131 Gemfile dependencies, 260 gems now installed. rails-default-1 | Gems in the groups 'development' and 'test' were not installed. rails-default-1 | Bundled gems are installed into /gems rails-default-1 | 1 installed gem you directly depend on is looking for funding. rails-default-1 | Run bundle fund for details rails-default-1 | + BUNDLE='bundle check' rails-default-1 | + bundle check rails-default-1 | The Gemfile's dependencies are satisfied rails-default-1 | + exec bundle exec rails s -p 3000 -b 0.0.0.0 rails-default-1 | => Booting Puma rails-default-1 | => Rails 7.0.8.7 application starting in production rails-default-1 | => Run bin/rails server --help for more startup options rails-default-1 | Exiting rails-default-1 | /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:146:in block in resolve_master': undefined method call' for an instance of Redis::Client (NoMethodError) rails-default-1 | rails-default-1 | host, port = sentinel_client.call("SENTINEL", "get-master-addr-by-name", @name) rails-default-1 | ^^^^^ rails-default-1 | Did you mean? call_v rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:188:in block in each_sentinel' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:184:in each' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:184:in each_sentinel' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:145:in resolve_master' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:137:in block in config' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:135:in synchronize' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:135:in config' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/sentinel_config.rb:81:in host' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/ruby_connection.rb:119:in connect' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/ruby_connection.rb:51:in initialize' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client.rb:746:in new' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client.rb:746:in block in connect' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client/middlewares.rb:12:in connect' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client.rb:745:in connect' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client.rb:732:in raw_connection' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client.rb:697:in ensure_connected' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-client-0.21.1/lib/redis_client.rb:292:in call_v' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-5.0.6/lib/redis/client.rb:73:in call_v' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-5.0.6/lib/redis.rb:167:in block in send_command' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-5.0.6/lib/redis.rb:166:in synchronize' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-5.0.6/lib/redis.rb:166:in send_command' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-5.0.6/lib/redis/commands/strings.rb:191:in get' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-namespace-1.10.0/lib/redis/namespace.rb:558:in wrapped_send' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-namespace-1.10.0/lib/redis/namespace.rb:515:in call_with_namespace' rails-default-1 | from /gems/ruby/3.3.0/gems/redis-namespace-1.10.0/lib/redis/namespace.rb:389:in block (2 levels) in <class:Namespace>' rails-default-1 | from /app/lib/global_config.rb:42:in block in load_from_cache' rails-default-1 | from /gems/ruby/3.3.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in block (2 levels) in with' rails-default-1 | from /gems/ruby/3.3.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in handle_interrupt' rails-default-1 | from /gems/ruby/3.3.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in block in with' rails-default-1 | from /gems/ruby/3.3.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in handle_interrupt' rails-default-1 | from /gems/ruby/3.3.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in with' rails-default-1 | from /app/lib/global_config.rb:42:in load_from_cache' rails-default-1 | from /app/lib/global_config.rb:20:in get_value' rails-default-1 | from /app/lib/chatwoot_app.rb:21:in chatwoot_cloud?' rails-default-1 | from /app/app/dashboards/account_dashboard.rb:17:in <class:AccountDashboard>' rails-default-1 | from /app/app/dashboards/account_dashboard.rb:3:in

' rails-default-1 | from /usr/local/lib/ruby/3.3.0/bundled_gems.rb:74:in require' rails-default-1 | from /usr/local/lib/ruby/3.3.0/bundled_gems.rb:74:in block (2 levels) in replace_require' rails-default-1 | from /gems/ruby/3.3.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/kernel.rb:26:in require' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/cref.rb:91:in const_get' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/cref.rb:91:in get' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:173:in block in actual_eager_load_dir' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/helpers.rb:47:in block in ls' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/helpers.rb:25:in each' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/helpers.rb:25:in ls' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:168:in actual_eager_load_dir' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:17:in block (2 levels) in eager_load' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:16:in each' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:16:in block in eager_load' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:10:in synchronize' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader/eager_load.rb:10:in eager_load' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader.rb:413:in block in eager_load_all' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader.rb:411:in each' rails-default-1 | from /gems/ruby/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/loader.rb:411:in eager_load_all' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/application/finisher.rb:74:in block in module:Finisher' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:32:in instance_exec' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:32:in run' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:61:in block in run_initializers' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:231:in block in tsort_each' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:353:in block (2 levels) in each_strongly_connected_component' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:434:in each_strongly_connected_component_from' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:352:in block in each_strongly_connected_component' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:350:in each' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:350:in call' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:350:in each_strongly_connected_component' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:229:in tsort_each' rails-default-1 | from /usr/local/lib/ruby/3.3.0/tsort.rb:208:in tsort_each' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:60:in run_initializers' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/application.rb:372:in initialize!' rails-default-1 | from /app/config/environment.rb:5:in <main>' rails-default-1 | from config.ru:3:in require_relative' rails-default-1 | from config.ru:3:in block in <main>' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/builder.rb:116:in eval' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/builder.rb:116:in new_from_string' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/builder.rb:105:in load_file' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/builder.rb:66:in parse_file' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/server.rb:349:in build_app_and_options_from_config' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/server.rb:249:in app' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/server.rb:422:in wrapped_app' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/server.rb:312:in block in start' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/server.rb:379:in handle_profiling' rails-default-1 | from /gems/ruby/3.3.0/gems/rack-2.2.14/lib/rack/server.rb:311:in start' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/commands/server/server_command.rb:38:in start' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/commands/server/server_command.rb:143:in block in perform' rails-default-1 | from <internal:kernel>:90:in tap' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/commands/server/server_command.rb:134:in perform' rails-default-1 | from /gems/ruby/3.3.0/gems/thor-1.3.1/lib/thor/command.rb:28:in run' rails-default-1 | from /gems/ruby/3.3.0/gems/thor-1.3.1/lib/thor/invocation.rb:127:in invoke_command' rails-default-1 | from /gems/ruby/3.3.0/gems/thor-1.3.1/lib/thor.rb:527:in dispatch' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/command/base.rb:87:in perform' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/command.rb:48:in invoke' rails-default-1 | from /gems/ruby/3.3.0/gems/railties-7.0.8.7/lib/rails/commands.rb:18:in <main>' rails-default-1 | from /usr/local/lib/ruby/3.3.0/bundled_gems.rb:74:in require' rails-default-1 | from /usr/local/lib/ruby/3.3.0/bundled_gems.rb:74:in block (2 levels) in replace_require' rails-default-1 | from /gems/ruby/3.3.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' rails-default-1 | from bin/rails:4:in `
'

rails-default-1 exited with code 0

To Reproduce

  1. Install Docker on your VM

  2. edit .env to use redis sentinel parameters: REDIS_SENTINELS=ip1:26379,ip2:26380,ip3:26381 REDIS_SENTINEL_MASTER_NAME=mymaster REDIS_PASSWORD=redis REDIS_SENTINEL_PASSWORD=redis

  3. Prepare the database docker compose run --rm rails bundle exec rails db:chatwoot_prepare

  4. build and start the service docker compose up -d --build

Expected behavior

app starts and works with redis sentinel without errors

Environment

Docker

Cloud Provider

None

Platform

Browser

Operating system

No response

Browser and version

No response

Docker (if applicable)

No response

Additional context

No response

hazem-nasser96 avatar Jun 04 '25 01:06 hazem-nasser96

The key error causing your Rails container to crash and restart repeatedly is: NoMethodError: undefined method call' for an instance of Redis::Client

redis-client 0.22.x changed the API In recent versions of redis-client, Redis::Client no longer responds to .call. Instead, you must use .call_v

This means some code is calling the Redis Sentinel client with the old call(...) method, which no longer exists.

So we need to change it to call_v

hazem-nasser96 avatar Jun 04 '25 01:06 hazem-nasser96

I have the same problem and I don't know how to solve it.

Image

REDIS_URL=redis://mymaster REDIS_PASSWORD=Abc123 REDIS_SENTINELS=redis-sentinel-01:26379,redis-sentinel-02:26379,redis-sentinel-03:26379 REDIS_SENTINEL_MASTER_NAME=mymaster REDIS_SENTINEL_PASSWORD=Abc123

ScraaaxA-Br avatar Jun 05 '25 02:06 ScraaaxA-Br

Any luck solving this????

azeemuddinahmad avatar Oct 08 '25 10:10 azeemuddinahmad

Any luck solving this????

No. Use Redis Standalone.

ScraaaxA-Br avatar Oct 08 '25 13:10 ScraaaxA-Br