thingsboard-pe-k8s
thingsboard-pe-k8s copied to clipboard
Unable to deploy tb-integration-executor with redis sentinel
Hello, i'm facing an issue with deployment of tb-integration-executor and redis sentinel. I was able to deploy tb-node with sentinel configured (even tho some important information is missing in the TB Core component docs) with the following StatefulSet env vars:
- name: CACHE_TYPE
value: "redis"
- name: REDIS_CONNECTION_TYPE
value: "sentinel"
- name: REDIS_MASTER
value: "sts-redis-master"
- name: REDIS_SENTINELS
value: "sts-redis-node-0.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-1.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-2.sts-redis-headless.redis.svc.cluster.local:26379"
- name: REDIS_SENTINEL_PASSWORD
value: <Password>
- name: REDIS_PASSWORD
value: <Password>
So after seeing that the tb-integration-executor also specifies redis as a parameters in the minikube deployment yaml files. I thought i could just copy this configuration and it will work. But to my surprise it didn't. When is set my thingsboard.yaml for the integration-executor env vars like this:
- name: CACHE_TYPE
value: "redis"
- name: REDIS_CONNECTION_TYPE
value: "sentinel"
- name: REDIS_MASTER
value: "sts-redis-master"
- name: REDIS_SENTINELS
value: "sts-redis-node-0.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-1.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-2.sts-redis-headless.redis.svc.cluster.local:26379"
- name: REDIS_SENTINEL_PASSWORD
value: <Password>
- name: REDIS_PASSWORD
value: <Password>
the pod logs look like this :
thingsboard-tb-integration-executor-0-1725375316634848100.log
Caused by: java.lang.IllegalArgumentException: Sentinel Master Id must not be null or empty
at org.springframework.util.Assert.hasText(Assert.java:289)
at org.springframework.data.redis.connection.SentinelMasterId.<init>(SentinelMasterId.java:34)
at org.springframework.data.redis.connection.RedisConfiguration$SentinelConfiguration.setMaster(RedisConfiguration.java:357)
at org.thingsboard.server.cache.TBRedisSentinelConfiguration.loadFactory(TBRedisSentinelConfiguration.java:65)
at org.thingsboard.server.cache.TBRedisCacheConfiguration.redisConnectionFactory(TBRedisCacheConfiguration.java:105)
at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0.CGLIB$redisConnectionFactory$31(<generated>)
at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0$$FastClassBySpringCGLIB$$4055d0d5.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0.redisConnectionFactory(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 56 common frames omitted
So after looking in some thingsboard code i found that the key to provide redis sentinel master name is: redis.sentinel.master So i thought of env variable would look like this: REDIS_SENTINEL_MASTER, when i tried that in the env vars for the integration-executor i got the following logs:
thingsboard-tb-integration-executor-0-1725375710414817500.log
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is sts-redis-master master is running...
at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:293)
at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:217)
at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:415)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:398)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
... 52 common frames omitted
Please bear in mind that this redis sentinel cluster is working fine for the tb-node. Any help is appreciated, Thank you Sam