WatchTree Channel is Closed Error after Implementing Redis Server Provider Authentication

[svatsala]

Welcome!

• [x] Yes, I've searched similar issues on GitHub and didn't find any.
• [x] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

We have a traefik proxy pod that should connect to our redis instance pods in the same kubernetes cluster. We implemented auth on our redis pods so our traefik.toml looks below. We found that traefik with redis works if I disable auth on redis.

What did you see instead?

However, after we've implemented redis auth, we are getting the following looping error in our traefik proxy that causes our configuration to reload. Anyone know why this might be? Is this an issue on the redis or traefik side? Has anyone seen it before?

What version of Traefik are you using?

redis version: 7.4.1 traefik version: 3.2.0

What is your environment & configuration?

traefik.toml:

# enable the api
    [api]
    [accessLog]
    [ping]
    entrypoint = "ping"
    # the public port where traefik accepts http requests
    [entryPoints.http]
    address = ":{{ $.Values.proxy.service.internalPort }}"
    # the port on localhost where the traefik api should be found
    [entryPoints.auth_api]
    address = ":{{ $.Values.proxy.traefik.api.internalPort }}"
    [entryPoints.ping]
    address = ":{{ $.Values.proxy.traefik.probeInternalPort }}"
    [log]
    level = "DEBUG"

    [providers.redis]
    username = "<username>"
    password = "<pwd>"
    # the Redis address
    endpoints = ["nbt-redis-sentinel.{{ $.Release.Namespace }}.local-cluster.local-dc.com:{{ $.Values.proxy.redis.sentinel.internalPort }}"]
    # the prefix to use for the static configuration
    rootKey = "traefik"

    [providers.redis.sentinel]
    masterName = "redis-ha"

If applicable, please paste the log output in DEBUG level

>>>2024-11-14T13:50:51Z DBG github.com/traefik/traefik/v3/pkg/provider/kv/storewrapper.go:60 > WatchTree: traefik
>>>2024-11-14T13:50:51Z DBG github.com/traefik/traefik/v3/pkg/provider/kv/storewrapper.go:78 > List: traefik
>>>2024-11-14T13:50:51Z ERR github.com/traefik/traefik/v3/pkg/provider/kv/kv.go:127 > Provider error, retrying in 5.062710016s error="the WatchTree channel is closed" providerName=redis

[nmengin]

Hello @svatsala,

Thanks for reaching out.

Could you provide a minimal reproducible case (for instance, full Kubernetes manifest to reproduce the issue)? In the meantime time, if any community member can help us find verified steps to reproduce and fix the issue if possible, we would love the help.

[Anant-2005]

Hey, Can I get assigned for this issue?

[svatsala]

What type of manifest? Service, pod, or deployment? Separately, I found that the error originates from this line but not sure why

[Aleksei-Poliakov]

I had "the WatchTree channel is closed" in the logs all the time as well, turned out the Redis user the Traefik was configured with had the following permissions "on ~* -@all +@read" - which isn't sufficient for keyspace event notifications to work; after adding an extra scope "on ~* -@all +@read +@pubsub" the error went away (I also had to enable notify-keyspace-events on our AWS ElastiCache Redis instance)

[nmengin]

Hey @Aleksei-Poliakov ,

Thank you for the feedback.

I close the issue. Please feel free to reach out if the issue reappears, and we'll reopen it.