[loki-distributed] official docs not working for memcached

[rotarur]

I see some docs related configurations not working in loki-distributed.

in the official docs https://grafana.com/docs/loki/latest/configuration/ is documented memcached_client with service memcached, but it seems not to work when configuring in loki-distributed.

Also I'm seeing in this readme https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed that for memcached_client.service we mush use http.

Which one to use? Cause both are failing

[rotarur]

anyone?

[alexandre1984rj]

I'm having the same issue. None of them are working http or memcached. I also have tried with memcache as explicited here: https://github.com/grafana/loki/issues/3070

[sureshgoli25]

We also tried to use memcache but memcache not getting used nor we see logs in memcache.

[ekolganov]

@rotarur `
service: memcached-client

`

[zanhsieh]

@rotarur @alexandre1984rj @sureshgoli25 @karamel32 Would you mind to open the same issue here: https://github.com/grafana/loki/issues and link back to this issue?

[thiDucTran]

for chart loki-distributed-0.71.2, I was getting logs like below

level=warn ts=2023-08-25T19:28:14.064576809Z caller=memcached_client.go:228 msg="error updating memcache servers" err="lookup _memcached._tcp.loki-loki-distributed-memcached-frontend on 10.0.0.10:53: no such host"

I had to change the port of service ***-loki-distributed-memcached-frontend from memcached-client to memcached like below

apiVersion: v1
kind: Service
metadata:
  name: loki-loki-distributed-memcached-frontend
  namespace: default
  labels:
    app.kubernetes.io/name: loki-distributed
    app.kubernetes.io/instance: loki
    app.kubernetes.io/component: memcached-frontend
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: memcached
      port: 11211
      targetPort: http
      protocol: TCP
    - name: http-metrics
      port: 9150
      targetPort: http-metrics
      protocol: TCP
  selector:
    app.kubernetes.io/name: loki-distributed
    app.kubernetes.io/instance: loki
    app.kubernetes.io/component: memcached-frontend

[dmitry-mightydevops]

my 5c if you installed memcached from bitnami and use grafana/loki helm chart.

➜ kgsvc | grep memcached
loki-memcached              ClusterIP   172.20.255.173   <none>        11211/TCP           18h

the value you want to put into loki.memcached.{chunk_cache, results_cache}.service is the name of the port of the service:

➜ kgsvcn loki loki-memcached -o yaml | k neat | yq '.spec.ports[0].name'
memcache

final:

loki:
      auth_enabled: false
      storage:
        bucketNames:
          admin: prod-eks-loki-admin
          chunks: prod-eks-loki-chunks
          ruler: prod-eks-loki-ruler
        type: s3
        s3:
          region: us-east-1
      memcached:
        chunk_cache:
          enabled: true
          host: loki-memcached.loki.svc
          service: memcache
          batch_size: 256
          parallelism: 10
        results_cache:
          enabled: true
          host: loki-memcached.loki.svc
          service: memcache
          timeout: 500ms
          default_validity: 12h

then you should be able to get rid of these messages in loki namespace:

{namespace="loki"} |= "memcached"

test memcached via CLI:

k run --image saritasallc/network-multitool utils --overrides='{"apiVersion": "v1", "spec": {"nodeSelector": { "ops": "true"}}}' -- sleep 3000

✗  k exec -ti utils -- sh                                                                                                                        
/ # telnet loki-memcached.loki.svc 11211
Connected to loki-memcached.loki.svc
stats slabs
STAT 4:chunk_size 192
STAT 4:chunks_per_page 5461
STAT 4:total_pages 1
STAT 4:total_chunks 5461
STAT 4:used_chunks 112
STAT 4:free_chunks 5349
STAT 4:free_chunks_end 0
STAT 4:get_hits 568
STAT 4:cmd_set 112
STAT 4:delete_hits 0
STAT 4:incr_hits 0
STAT 4:decr_hits 0
STAT 4:cas_hits 0
STAT 4:cas_badval 0
STAT 4:touch_hits 0
STAT 5:chunk_size 240

and queries caching started working - the same query reduced from 3s to 100ms.