[loki] msg="error running loki" err="mkdir /rules: read-only file"

[waterdrop01]

Hello,

Here is my loki-values.yml file:

persistence:
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 20Gi
  annotations: {}
  storageClassName:

updateStrategy:
  type: RollingUpdate
# updateStrategy:
#   type: Recreate

config:
  compactor:
    working_directory: /data/loki/boltdb-shipper-compactor
    shared_store: filesystem
  schema_config:
    configs:
      # - from: 2018-04-15
      #   store: boltdb
      #   object_store: filesystem
      #   schema: v9
      #   index:
      #     prefix: index_
      #     period: 168h
      - from: 2020-12-05
        store: boltdb-shipper
        object_store: filesystem
        schema: v11
        index:
          prefix: index_
          period: 24h
  storage_config:
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
      shared_store: filesystem
    filesystem:
      directory: /data/loki/chunks
  ruler:
    storage:
      type: local
      local:
        directory: /rules/fake
    rule_path: /tmp/scratch
    alertmanager_url: http://prometheus-alertmanager.monitoring
    ring:
      kvstore:
        store: inmemory
    enable_api: true

And here is the loki error message when installing the chart using helm:

level=info ts=2021-07-22T21:23:22.794720884Z caller=main.go:130 msg="Starting Loki" version="(version=2.2.1, branch=HEAD, revision=babea82ef)"
level=error ts=2021-07-22T21:23:22.794935078Z caller=log.go:106 msg="error running loki" err="mkdir /rules: read-only file system\nerror initialising module: ruler-storage\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:105\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:75\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:220\nmain.main\n\t/src/loki/cmd/loki/main.go:132\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"

It seems related to this issue https://github.com/grafana/loki/issues/3005

Thanks!

[shinebayar-g]

If you're enabling ruler you have to supply directories, take a look at my issue.

[waterdrop01]

Thanks @shinebayar-g for your reply.

We are using two different charts. Mine is [loki] but the one you encountered an issue with is [loki-distributed].

Their values.yml specs are different. For instance [loki] does not have "enable" or "directories" keys.

So adding a directories key result in another error (directories not found).

[shinebayar-g]

Oh didn't notice that sorry.

[dpiessens]

So I believe I know what's going on: the helm chart mounts /rules as a ConfigMap volume. While it is mounted as read/write I don't believe it can be saved to, especially if sub-directories are involved. My assumption was rules created by the API (Grafana) would be written to /tmp/scratch. I think the answer might be to mount the configmap as a sub-directory and the root /rules as an empty dir, similar to what @shinebayar-g said.

[MikeMichel]

This error also pops up when you did not specify the alerting_groups: part.

[qdupuy]

@MikeMichel Example ?

[MikeMichel]

Check https://github.com/grafana/helm-charts/blob/main/charts/loki/values.yaml#L272

[qdupuy]

ah, it's for loki and not loki-distributed

[joenzx]

I have the same problem. Is there a solution? @waterdrop01

[timam]

I have the same problem.

[ognjenVlad]

extraVolumes:
  - name: rules
    emptyDir: {}
extraVolumeMounts:
  - name: rules
    mountPath: /rules

Fixed it for me

[genieai-vikas]

@ognjenVlad I am facing the same issue. Where have you added this?

[ognjenVlad]

@genieai-vikas Check chart values, you should find it in there

[abvijaykumar]

Did anybody find a solution to this. I am struggling with this problem, I tried all that is litedn here, I am still getting the following error

level=error ts=2022-08-18T08:10:20.15790142Z caller=log.go:103 msg="error running loki" err="mkdir /etc/loki/rules: read-only file system\nerror initialising module: ruler-storage\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:341\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1133"

Here is the values file I am using. Please check and let me know if I am doing any mistakes

config:
  ruler:
    storage:
      type: local
      local:
        directory: /etc/loki/rules
    rule_path: /tmp/loki/rules-temp
    alertmanager_url: http://alertmanager-operated:9093
    ring:
      kvstore:
        store: inmemory
    enable_api: true
    enable_alertmanager_v2: true

serviceMonitor:
  enabled: true
  interval: "1m"
  prometheusRule:
    enabled: true
    rules:
    - alert: bozo-book-lib-errors
      expr: "{namespace=\"bozo-book-library-dev\"} |= `error`"
      for: 2m
      annotations:
        message: 'Error ({{ $value }}).'

extraVolumes:
  - name: rules
    emptyDir: {}
extraVolumeMounts:
  - name: rules
    mountPath: /rules

[Demonsthere]

For those who will run into the same wall: Loki needs 2 volumes/volumeMouns. One, as working directory in which it has full write permissions, and a second from which it reads supplied rules.

For the read container:

      extraVolumes:
        - name: loki-rules
          configMap:
            name: loki-alertinggroups
        - name: loki-rules-generated
          emptyDir: {}
      extraVolumeMounts:
        - name: loki-rules-generated
          mountPath: /rules
        - name: loki-rules
          mountPath: /var/loki/rules

[K890]

Loki is trying to write the WAL to its default location which is not writable (should be fixed in the helm chart) To fix this you need to write to a location that is writable by the pod. So either you can mount an extra emtyDir and point the wal to that directory or if you are already using a PVC for ruler, just change the wal to point to a directory within the PVC. By default the PVC is mounted to /var/loki so I am using /var/loki/wal as my wal directory and it works.

ruler:
  enabled: true
  kind: StatefulSet
  persistence:
    enabled: true
loki:
  config: |
    ruler:
      wal:
        /var/loki/wal  

Note: Kept only the conf that is relevant in the current context.