redis_exporter
redis_exporter copied to clipboard
Connection Reset by Peer upon a "Cluster Refresh"
I have a redis 6.2.6. cluster over 3 VMs, with 2 nodes per VM (ports 7005, 7006) Redis exporter is running with following command
./redis_exporter --redis.addr=redis://git-kfkrds01:7005 --skip-tls-verification --tls-ca-cert-file=/apps/Testing/redistest/ca/ca.crt --tls-client-key-file=/apps/Testing/redistest/ca/redis.key --tls-client-cert-file=/apps/Testing/redistest/ca/redis.crt --set-client-name=true --is-cluster=true
Prometheus extracts info with
- job_name: 'redis_exporter_targets'
static_configs:
- targets:
- rediss://git-kfkrds01:7005
- rediss://git-kfkrds01:7006
- rediss://git-kfkrds02:7005
- rediss://git-kfkrds02:7006
- rediss://git-kfkrds03:7005
- rediss://git-kfkrds03:7006
metrics_path: /scrape
scheme: http
tls_config:
insecure_skip_verify: true
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: git-kfkrds01:9121
## config for scraping the exporter itself
- job_name: 'redis_exporter'
static_configs:
- targets:
- git-kfkrds01:9121
Grafana dashboards show actual exported metrics.
However the redis-exporter output contains clauses like
ERRO[2880] Couldn't set client name, err: read tcp 10.x.x.93:46436->10.x.x.93:7005: read: connection reset by peer
ERRO[2880] Redis INFO err: set tcp 10.x.x.93:46436: use of closed network connection
ERRO[2880] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:33556->10.x.x.95:7006: read: connection reset by peer
ERRO[2883] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:46576->10.x.x.95:7005: read: connection reset by peer
ERRO[2885] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:52606->10.x.x.94:7005: read: connection reset by peer
ERRO[2888] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:55820->10.x.x.93:7006: read: connection reset by peer
ERRO[2889] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:37098->10.x.x.94:7006: read: connection reset by peer
ERRO[2892] Cluster refresh failed: redisc: all nodes failed
read tcp 10.x.x.93:47072->10.x.x.93:7005: read: connection reset by peer
When enabled --debug, I get following additional info.
DEBU[0001] c.Do() - running command: LATENCY [LATEST]
DEBU[0001] c.Do() - done
DEBU[0001] Load key pair: /apps/Testing/redistest/ca/redis.crt /apps/Testing/redistest/ca/redis.key
DEBU[0001] Load CA cert file: /apps/Testing/redistest/ca/ca.crt
DEBU[0001] Creating cluster object
DEBU[0001] Running refresh on cluster object
ERRO[0001] Cluster refresh failed: redisc: all nodes failed
read tcp 10.244.226.93:44172->10.244.226.94:7006: read: connection reset by peer
DEBU[0001] Creating redis connection object
DEBU[0001] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0001] keys: []exporter.dbKeyPair(nil)
DEBU[0001] parseKeyArg(): Got empty key arguments, parsing skipped
DEBU[0001] e.singleKeys: []exporter.dbKeyPair(nil)
DEBU[0001] e.keys: []exporter.dbKeyPair(nil)
How can I avoid such error logging?