go-redis/v9: incr, ping commands getting EOF constantly after about a day of successful periodic (once in few seconds) calls

[chitturs]

Issue tracker is used for reporting bugs and discussing new features. Please use stackoverflow for supporting issues.

Client: GoLang using redisv9 Server: Azure Redis Cache

Client reads messages from Azure service bus and then updates the Redis Cache using Incr(). This is repeated ad infinitum. Everything works for about a day. The calls are made once in few seconds. After about a day, Incr() returns EOF and never succeeds. At that point, ping() also returns EOF.

Expected Behavior

No failures in Incr() or ping().

Current Behavior

Incr() and ping() fail with EOF.

Possible Solution

It is unclear what the problem is. Suspicion is a broken connection. The connection should never be idle for the idle timeout to kick in. The server has an idle timeout of 10 minutes.

Steps to Reproduce

import (
	  "context"
	  "crypto/tls"
	  "errors"
	  "fmt"
	  "os"
	  "strings"
	  "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
	  "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	  "github.com/Azure/azure-sdk-for-go/sdk/messaging/azservicebus"
	  "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/redis/armredis/v3"
	  "github.com/redis/go-redis/v9"
)
	  op := &redis.Options{Addr: redisHostName + ":" + fmt.Sprint(*getRes.Properties.SSLPort), Username: msiObjectId, Password: accessToken.Token, TLSConfig: &tls.Config{MinVersion: tls.VersionTLS12}}

client := redis.NewClient(op)

	for _, message := range messages {
		body := message.Body
		redisCacheKey := strings.Trim(string(body), `"`)
		result, err := redisclient.Incr(ctx, redisCacheKey).Result()
		if err != nil {
			if errors.Is(err, context.Canceled) {
				fmt.Println("context was canceled while incrementing cache, return")
				return
			}
			**fmt.Println(fmt.Errorf("redis cache increment failed for key %s, err %v", redisCacheKey, err))**

			// Check if the connection is still alive, https://redis.io/commands/ping/.
			pong, err := redisclient.Ping(ctx).Result()
			fmt.Println(fmt.Errorf("redis cache ping response %s, err %v, timeouts %d", pong, err, redisclient.PoolStats().Timeouts))
			continue
		}

		fmt.Println("Key:", redisCacheKey, "Result:", result)
	}

Context (Environment)

My AKS cluster app is partially broken in functionality. The Redis Cache is used for stats purposes of other activities in the cluster. The stats is now not working after about a day.