no rule groups found when using S3 and configMap to load rules

[yelhouti]

Describe the bug I'm trying to load rules using configMap while using s3 and ssd . I can see that the sidecar copying rule files into /rules (I even tried changing config to have them copied to /loki/rules). Querying the rules always returns: no rule groups found

To Reproduce Steps to reproduce the behavior:

1. Started Loki 3.0.0
2. Started Promtail 2.9.3

values:

loki:
  auth_enabled: false 
  storage:
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    type: s3
    s3:
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyId: loki
      s3ForcePathStyle: true
      insecure: true
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: index_
          period: 24h

curl http://loki-backend:3100/api/prom/rules
# or
curl http://loki-gateway/api/prom/rules

Expected behavior To see the rules.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.

[JStickler]

Questions have a better chance of being answered if you ask them on the community forums.

[edwlarkey]

I'm having a similar problem to this using the helm chart. I am not able to provision recording rules in the ruler section.

loki:
  storage:
    bucketNames:
      chunks: REPLACEME
      ruler: REPLACEME
      admin: REPLACEME
    type: s3
    s3:
      region: REPLACEME
      s3ForcePathStyle: true
  rulerConfig:
    wal:
      dir: /var/loki/ruler-wal
    rule_path: /etc/loki/rules
    remote_write:
      enabled: true
      clients:
        mimir:
          url: http://mimir-nginx.monitoring.svc.cluster.local/api/v1/push
ruler:
  replicas: 1
  nodeSelector:
    monitoring-worker: "true"
  persistence:
    enabled: true
  directories:
    fake:
      rules.yaml: |
        groups:
        - name: api
          rules:
            - record: api:loglevel:sum_rate
              expr: sum by(level) (rate({container="api"} [10m]))

The ruler logs:

level=info ts=2024-05-22T16:22:21.599510548Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:23.445419254Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:33.81113425Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:41.711968209Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:43.424880297Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:53.69105127Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:02.140777628Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:03.763753726Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:13.713900816Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:22.022359636Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake

[yelhouti]

You can not use both configMaps and s3. I recommand configMaps for gitops... to use configMaps instead of s3:

  rulerConfig:
    storage:
      type: local
      local:
        directory: /var/loki/rules
    rule_path: /var/loki/rules-temp
    alertmanager_url: http://kube-prometheus-stack-alertmanager.prometheus.svc:9093
    enable_alertmanager_v2: true
    wal:
      dir: /var/loki/ruler-wal

If you are deploying on default mode (SSD).

IMO this should be better documented. Thanks for the person who helped me figure it out.

[edwlarkey]

I suspected that, but I couldn't figure out how to turn off S3 just for the ruler. Using your config to override the storage config for ruler worked. Thank you!

[jimmy-ungerman]

@yelhouti so you aren't using s3 to store your rules now? s3 for the loki object storage, but local filesystem using configmaps for the rules storage?

[yelhouti]

Yes I m using configmaps now, which I think is better

[jimmy-ungerman]

Do you mind posting your values.yaml as much as you can? I'm imagining you're creating the configmap using .Values.loki.extraObjects to create rules configmaps, then the .Values.sidecar.rules.enabled.true to copy those configmaps into your rules directory?

Just want to make sure I understand correctly

[nlamirault]

I think it is same as https://github.com/grafana/loki/issues/11508

[MartinEmrich]

I have the same issue, but I am confused now.

Isn't the ruler storage also where all the time series for Recording Rules and various state is stored? That would most certainly not work with "local" and a mounted config map. I would want that on a fast block device (local).

Also I did not expect the Loki helm chart to use S3 for the ruler. I run in Simple-Scalable mode, the ruler runs within the "backend" StatefulSet which has a set of PersistentVolumeClaims (i.e. block storage based filesystem volumes).

As I use S3 for the actual logs, I had to configure loki.storage with a ruler bucket name:

        s3:
          region: my-region-1
        bucketNames:
          chunks: my-loki-logs
          ruler: my-loki-logs
          admin: my-loki-logs

If I would remove the "ruler" entry there, the helmchart would just render bucketNames: null, making the backend crash.

How can I have it use S3 for the actual logs, but not for the ruler? I fear that just overriding the rulerConfig.storage object is not enough?