[loki-distributed] Configuring loki-distributed unusual error messages

[Luwdo]

It appears to be working, I see logs in aws s3 and grafana is allowing me to search them.

However a number of services are printing out errors that I am trying to determine if it is a misconfiguration or normal behavior.

The Querier has these

level=info ts=2021-11-12T21:17:27.667545788Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:30.676734868Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:33.680465065Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:36.686333708Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:39.688430958Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:42.691831667Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:45.694758692Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:48.697076431Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:51.700368264Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:54.706555806Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"
level=info ts=2021-11-12T21:17:57.710540193Z caller=dns_resolver.go:209 msg="failed DNS SRV record lookup" err="lookup _grpclb._tcp.wsm-loki-loki-distributed-query-frontend on 172.20.0.10:53: no such host"

The Distributor has these:

ts=2021-11-12T18:27:16.478409182Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve wsm-loki-loki-distributed-memberlist: lookup wsm-loki-loki-distributed-memberlist on 172.20.0.10:53: no such host"
ts=2021-11-12T18:27:18.411813805Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve wsm-loki-loki-distributed-memberlist: lookup wsm-loki-loki-distributed-memberlist on 172.20.0.10:53: no such host"
ts=2021-11-12T18:27:22.356667302Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve wsm-loki-loki-distributed-memberlist: lookup wsm-loki-loki-distributed-memberlist on 172.20.0.10:53: no such host"
ts=2021-11-12T18:27:27.515471894Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve wsm-loki-loki-distributed-memberlist: lookup wsm-loki-loki-distributed-memberlist on 172.20.0.10:53: no such host"
ts=2021-11-12T18:27:42.115053349Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve wsm-loki-loki-distributed-memberlist: lookup wsm-loki-loki-distributed-memberlist on 172.20.0.10:53: no such host"

This is loki 2.4.1 deployed on aws eks self-managed cluster configuration.

[Luwdo]

They do no not appear to be stoping logs from being processed

[benjyiw]

I just installed Loki via the loki-distributed chart and started seeing these errors. Looks like this issue is a duplicate of #801, of which there is already a PR open with a suggested fix #803.

[Luwdo]

Forked the repo and installed the chart locally with adding the new named service port for grpc for the query frontend.

Will keep an eye on https://github.com/grafana/helm-charts/pull/803 for geting back to mainline as the right fix is being debated.

[korenlev]

see the same issues with 2.4.2

[LockedThread]

^

[LinTechSo]

Hi. any updates ? see the same issues with 2.6.0

[patsevanton]

Same issue 2.6.1 - helm chart loki-distributed-0.65.3

[zanhsieh]

Could you guys raise this error here?

https://github.com/grafana/loki/issues

[patsevanton]

This error on deploy helm chart. Need wait for run all component loki.

Sorry. I wait and have error.

[ankitmahajan507]

Hi, Can someone update on this. Or share the fix for this.

[nicocrib]

Adding this lines index_gateway_client: server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095 in the config: storage_config: section in the values file, after {{- if .Values.indexGateway.enabled}} solved the problem in my case.

config:
  storage_config:
      {{- if .Values.indexGateway.enabled}}
      index_gateway_client:
          server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095
      {{- end}}

[karr1gan]

could you please help to solve the problem?