REDIS Cluster Mode Fails To Connect With TLS

[ersinors]

Describe the bug I have an AWS ElasticCache Serverless Instance. I can connect with 'redis-cli' but n8n can not connect and gives the following error :

Error with Redis: 
n8n_1  | Redis unavailable - trying to reconnect...
n8n_1  | [ioredis] Unhandled error event: ClusterAllFailedError: Failed to refresh slots cache.
n8n_1  |     at tryNode (/usr/local/lib/node_modules/n8n/node_modules/ioredis/built/cluster/index.js:308:31)
n8n_1  |     at /usr/local/lib/node_modules/n8n/node_modules/ioredis/built/cluster/index.js:325:21
n8n_1  |     at /usr/local/lib/node_modules/n8n/node_modules/ioredis/built/cluster/index.js:662:24
n8n_1  |     at run (/usr/local/lib/node_modules/n8n/node_modules/ioredis/built/utils/index.js:117:22)
n8n_1  |     at tryCatcher (/usr/local/lib/node_modules/n8n/node_modules/standard-as-callback/built/utils.js:12:23)
n8n_1  |     at /usr/local/lib/node_modules/n8n/node_modules/standard-as-callback/built/index.js:33:51
n8n_1  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)

To Reproduce Steps to reproduce the behavior:

1. Create AWS ElasticCache Serverless.
2. Create EC2 Instance.
3. Update security inbound and outbound rules.
4. Install docker-compose and docker related stuff.
5. Use this docker-compose file :

version: "3.7"

services:
  n8n:
    image: docker.n8n.io/n8nio/n8n
    restart: always
    ports:
      - "80:5678"
    env_file:
      - ./.env
    environment:
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
    volumes:
      - n8n_data:/home/node/.n8n
      - /root/n8n:/files

volumes:
  n8n_data:

With following .env file in the same directory :

QUEUE_BULL_REDIS_TLS=true
QUEUE_BULL_REDIS_CLUSTER_NODES=redacted:6379

Expected behavior N8N should work.

Environment (please complete the following information):

• OS: Ubuntu Linux 22.04
• n8n Version latest.
• Database system postgresql.
• Operation mode queue.

[Joffcom]

Hey @ersinors,

Is the serverless redis instance running in cluster mode? I know normal redis instances on AWS works so I suspect this will be something specific with the serverless option.

[aeb-dev]

@Joffcom, yes, serverless redis works in cluster mode. However, n8n have configuration for redis cluster.

e.g https://github.com/n8n-io/n8n/blob/da1fe44d5246848e2ba7bb8bc5f4577685fbcee0/packages/cli/src/services/redis/RedisServiceHelper.ts#L116

https://github.com/n8n-io/n8n/blob/da1fe44d5246848e2ba7bb8bc5f4577685fbcee0/packages/cli/src/services/cache/redis.cache-manager.ts#L170

https://github.com/n8n-io/n8n/pull/6708

So I believe @ersinors meant that there is a bug somewhere that he can not connect to redis cluster

[Joffcom]

@aeb-dev we do indeed support clusters but I am not overly familiar with the serverless option but I do know someone had an issue with serverless on AWS before and using a normal redis cluster on AWS worked with no issues.

Based on this and a quick search online I suspect the issue could directly be linked to serverless redis, I will know more when I get a chance to test on Monday.

[ersinors]

Did you have a chance to test this?

[Joffcom]

@ersinors not fully, from what I have tested it works with Redis clusters it is only failing with Serverless which makes me think there could be a bit more to this.

I will get a dev ticket created shortly so this can be looked into in more detail. For now the best workaround would be to not the serverless option and use a standard Redis cluster.

[Joffcom]

Internal ticket for this is N8N-7185

[ersinors]

Can you share the link where I can follow the status of this ticket?

[Joffcom]

Hey @ersinors,

There is no public link at the moment, Once we have fixed it or if we have further questions one of us will pop a message on here to let you know.

[krynble]

@ersinors I was having a look at this issue and could not find any exact details in the docs, but I couldn't find any strong evidence that AWS Serverless Redis runs in cluster mode. It could just work with a usual connection.

Have you tried setting QUEUE_BULL_REDIS_HOST instead of the cluster config?

[krynble]

Just to give more context on my previous comment, from https://repost.aws/questions/QUDyEltSQJTsOu22rfUbhd5A/connect-to-elasticache-serverless-using-nodejs-ioredis it seems to use a regular new Redis instead of new Redis.Cluster.

Under the hood, this is exactly what n8n would do when not using Cluster mode as can be seen here as opposed to cluster mode here.

[aeb-dev]

@krynble AWS Serverless Redis uses cluster mode you can try to connect with redis-cli -c. I am also having this issue and I tried using QUEUE_BULL_REDIS_HOST but n8n fails

[krynble]

Thanks for the additional context @aeb-dev

I tried investigating this but unfortunately my knowledge about AWS Elasticache for Redis Serverless is limited. I spent some time setting it up but it turns out to be a bit more complex than I antecipated.

Is it possible for you to try the regular redis deployment instead of the serverless option? I think this is a simpler deployment that would probably work fine.

[krynble]

I'll be closing this issue for now as n8n is working with regular Redis and this is our requirement to run queue mode.

Since Redis is still well and kicking (like not deprecated) and easily available, we won't be tackling this at this time.

[aeb-dev]

Sorry for late reply,

Is it possible for you to try the regular redis deployment instead of the serverless option? I think this is a simpler deployment that would probably work fine.

Ofcourse it is possible to do that but I would prefer not to since serverless is much easier to manage.

I tried investigating this but unfortunately my knowledge about AWS Elasticache for Redis Serverless is limited. I spent some time setting it up but it turns out to be a bit more complex than I antecipated.

I am not sure what was the issue, if you can tell me what happened I might help you to set things up.