RedisInsight icon indicating copy to clipboard operation
RedisInsight copied to clipboard

[Bug]: Can't add redis database from sentinel "Something went wrong adding the sentinel database."

Open djeanprost opened this issue 2 years ago • 6 comments

Preconditions (Any important steps we need to know)

Steps to reproduce (How to reproduce what you found step by step) Open redis Insight Click on add redis database Enter sentinel information and gives sentinel password, then next List of redis db hadled by sentinel appears, enter a password then click add : I get "Something went wrong adding the sentinel database." If I try with an incorrect password, I get a message telling the password is not ok.

Actual behavior (A short description of what you found) I get "Something went wrong adding the sentinel database."

ERROR 2022-08-23 12:24:29,210 django.request Internal Server Error: /api/instance/ Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/local/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view return view_func(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view return self.dispatch(request, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch response = self.handle_exception(exc) File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception self.raise_uncaught_exception(exc) File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception raise exc File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch response = handler(request, *args, **kwargs) File "./redisinsight/core/views/instance.py", line 208, in post File "./redisinsight/core/views/instance.py", line 129, in _save_redis_instance File "./redisinsight/core/services/database/_routines.py", line 80, in _wrapped_add_db_func File "./redisinsight/core/services/database/_routines.py", line 776, in add_redis_database File "./redisinsight/core/services/database/_routines.py", line 826, in add_sentinel_monitored_db File "./redisinsight/core/services/database/_routines.py", line 393, in _add_sentinel_monitored_db File "./redisinsight/core/services/database/_routines.py", line 281, in _probe_other_sentinel_nodes_info File "./redisinsight/core/services/database/_routines.py", line 281, in <setcomp> KeyError: 'host'

Expected behavior (A short description of what you expected to find)

Screenshots (Paste or drag-and-drop a screenshot or a link to a recording)

Additional context (Operating system, version of RedisInsight, Redis database version, Redis module version, database type, connection type, logs, or any other information) RedisInsight running in container 1.12.1 target Sentinel and redis also runs in a other container

djeanprost avatar Aug 23 '22 12:08 djeanprost

Hey @djeanprost ,

Can you give more details of the type of redis and they way the database is architectured?

GnaneshKunal avatar Aug 23 '22 17:08 GnaneshKunal

hello @GnaneshKunal Don't know if it is the data you want, but here what I have. I manage a 3 nodes cluster of redis in ha only (no sharding). Thoses 3 redis nodes are handled by 3 sentinel nodes. Each sentinel manages 1 redis instance. Redis instances are password protected, either the sentinel. For my lab, those 2x3 nodes are deployed on the same host, using different ports of course. Each sentinel knows its redis instance by its public IP, as they are all deployed on docker, this is always the host IP that is used. Redis version is 6.2.x.

Do you need something more ? I hope this will help. If the stack trace I gave is not enough, I can provide more log if needed.

djeanprost avatar Aug 24 '22 06:08 djeanprost

I'm having difficulties using with it with redis sentinel as well, but I'm using it inside k8s operator (https://github.com/spotahome/redis-operator), password-less.

Here's my config for the redis-cluster:

apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: redis
spec:
  sentinel:
    replicas: 3
    resources:
      requests:
        cpu: 50m
      limits:
        memory: 100Mi
    customConfig:
      - "down-after-milliseconds 2000"
      - "failover-timeout 3000"
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: "app.kubernetes.io/component"
                    operator: In
                    values:
                      - "sentinel"
              topologyKey: topology.kubernetes.io/zone
  redis:
    customConfig:
      - "maxclients 1000"
      - "hz 10"
      - "timeout 0"
      - "min-replicas-max-lag 600"
      - "min-replicas-to-write 1"
      - "tcp-keepalive 300"
      - "hash-max-ziplist-entries 1024"
      - "maxmemory 480mb"
      - "maxmemory-policy volatile-lru"
    replicas: 2
    resources:
      requests:
          cpu: 100m
          memory: 500Mi
      limits:
        cpu: 1500m
        memory: 1200Mi
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: "app.kubernetes.io/component"
                    operator: In
                    values:
                      - "redis"
              topologyKey: topology.kubernetes.io/zone
    storage:
      persistentVolumeClaim:
        metadata:
          name: redis-data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1G

When I try to add using the service hostname and port 26379 with 'mymaster' auth, the system returns wrong credentials error. Then I entered the redis, added a user cron with mypass as password, full access user cron on #... ~* &* +@all then, this error appeared on the redisinsight log:

ERROR 2022-08-29 23:40:22,037 django.request Internal Server Error: /api/instance/
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "./redisinsight/core/views/instance.py", line 208, in post
  File "./redisinsight/core/views/instance.py", line 129, in _save_redis_instance
  File "./redisinsight/core/services/database/_routines.py", line 80, in _wrapped_add_db_func
  File "./redisinsight/core/services/database/_routines.py", line 776, in add_redis_database
  File "./redisinsight/core/services/database/_routines.py", line 826, in add_sentinel_monitored_db
  File "./redisinsight/core/services/database/_routines.py", line 393, in _add_sentinel_monitored_db
  File "./redisinsight/core/services/database/_routines.py", line 281, in _probe_other_sentinel_nodes_info
  File "./redisinsight/core/services/database/_routines.py", line 281, in <setcomp>
KeyError: 'host'

I was able to connect to redis using the address it shows back in the interface. As redisinsight is not really needed for the system health, I can change the Ip's each time the pod restart.

renatocron avatar Aug 30 '22 17:08 renatocron

@djeanprost and @renatocron

I have made a fix for the same.

Can you test the following build:

reedislabs/redisinsight:hotfix-sentinel-probe

Let me know if it works.

GnaneshKunal avatar Sep 25 '22 11:09 GnaneshKunal

Hello @GnaneshKunal With the version hotfix-sentinel-probe, it's ok for me. By the way, the docker image is now the double of the size. Quite a big fix :)

djeanprost avatar Sep 26 '22 13:09 djeanprost

@djeanprost

The docker image also has the latest changes which also supports ARM. Please use that image until the next release. We will be releasing the fix in the upcoming release.

GnaneshKunal avatar Sep 30 '22 08:09 GnaneshKunal

Hello @GnaneshKunal can you tell me if this open issue will be made available in a next release of the docker image please ?

djeanprost avatar Dec 20 '22 09:12 djeanprost

@djeanprost

The latest version of Redisinsight V1 already has this issue fixed.

GnaneshKunal avatar Dec 20 '22 09:12 GnaneshKunal

So this one contains the fix redislabs/redisinsight:1.13.1 , doesn't it ? I guess we can close this issue then :)

djeanprost avatar Dec 20 '22 09:12 djeanprost

confirmed, it's working on redislabs/redisinsight:1.13.1

image

It was not working on redislabs/redisinsight:hotfix-sentinel-probe tho

renatocron avatar Dec 20 '22 10:12 renatocron

RedisInsight v1 is no longer supported, please create a new issue or re-open this one if you have the same issue with RedisInsight v2.

ViktarStarastsenka avatar Jun 14 '23 12:06 ViktarStarastsenka