Ruler WAL does not respect common:path_prefix argument

[verejoel]

Describe the bug Loki ruler Write-ahead log is not created in a directory prefixed by the path_prefix variable.

To Reproduce Steps to reproduce the behavior:

1. Set the path_prefix variable in Loki.

common:
  path_prefix: /var/loki

3. Launch a ruler instance
4. Observe that the WAL directory in the config (localhost:3100/config) is not correctly set.

ruler:
  wal:
    dir: ruler-wal

Expected behavior WAL directory should be prefixed with the value in the path_prefix variable.

common:
  path_prefix: /var/loki
...
ruler:
  wal:
    dir: /var/loki/ruler-wal

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm, loki-distributed chart.

Screenshots, Promtail config, or terminal output

level=info ts=2022-10-10T11:09:25.931386536Z caller=loki.go:374 msg="Loki started"
level=info ts=2022-10-10T11:09:25.932092231Z caller=mapper.go:154 msg="updating rule file" file=/tmp/loki/scratch/fake/rules.txt
level=error ts=2022-10-10T11:09:25.933186623Z caller=instance.go:240 storage=registry manager=tenant-wal instance=fake msg="failed to initialize instance" err="error creating WAL: create dir: mkdir ruler-wal: read-only file system"
level=error ts=2022-10-10T11:09:25.93367732Z caller=manager.go:272 storage=registry manager=tenant-wal msg="instance stopped abnormally, restarting after backoff period" err="failed to initialize instance: error creating WAL: create dir: mkdir ruler-wal: read-only file system" backoff=1s instance=fake

Here the directory cannot be created as the root filesystem is read-only. The /var/loki directory is mounted as an empty directory, and would be writeable. But the path_prefix has not been used, leading to this error.

[the-elder]

same issue for me

[eviscares]

Same for me, led me on a wild goose chase for a while.

[daboom]

Same. Workaround is setting absolute path for ruler.wal.dir

[pikeas]

Two years later, would be great if this could be fixed!