Grafana fails to become healthy

[utzuro]

I have enabled Grafana service as shown in the docs, and it runs but fails imidiately because of the read-only file system in nix store (obviously on nix).

Here is my code:

{ config, lib, pkgs, ... }:
{
  options = {
    services.monitoring = {
      enable = lib.mkEnableOption "Enable grafana stack";
      package = lib.mkPackageOption pkgs "monitoring" { };
    };
  };
  config =
    let
      cfg = config.services.monitoring;
    in
    lib.mkIf cfg.enable {
      services.grafana.gf1 = {
        enable = true;
      };
    };
}

I get this error:

Error: x failed to connect to database: failed to create SQLite database file "/nix/store/1km88kkjcrdn3vybwvx74pvfaxrhc44f-grafana-12.0.0+security-01/share/
grafana/grafana.db": open /nix/store/1km88kkjcrdn3vybwvx74pvfaxrhc44f-grafana-12.0.0+security-01/share/grafana/grafana.db: read-only file system

It can't create database, because nix doesn't allow it. As a workaround I did the setup for DB myself, but grafana still tries to write something to the readonly filesystem.

New code with DB setup

{ config, lib, pkgs, ... }:
{
  options = {
    services.monitoring = {
      enable = lib.mkEnableOption "Enable grafana stack";
      package = lib.mkPackageOption pkgs "monitoring" { };
    };
  };
  config =
    let
      cfg = config.services.monitoring;
    in
    lib.mkIf cfg.enable {

      services.postgres.pg-grafana = {
        enable = true;
        listen_addresses = "127.0.0.127";
        initialScript.after = "CREATE USER root SUPERUSER;";
      };

      services.grafana.gf1 = {
        enable = true;
        extraConf.database = with config.services.postgres.pg-grafana; {
          type = "postgres";
          host = "${listen_addresses}:${builtins.toString port}";
          name = "postgres"; # database name
        };
      };

      settings.processes."gf1".depends_on."pg-grafana".condition = "process_healthy";
    };
}

And it fails with this error:

logger=secrets t=2025-09-05T10:35:29.559024542+09:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.vl

Error: x failed to create directory "/nix/store/1km88kkjcrdn3vybwvx74pvfaxrhc44f-grafana-12.0.0+security-01/share/grafana/png": mkdir /nix/store/
1km88kkjcrdn3vybwvx74pvfaxrhc44f-grafana-12.0.0+security-01/share/grafana/png: read-only file system

Looks like grafana package is not patched to be used within the nix ecosystem?

[shivaraj-bh]

The tests for grafana are passing on the latest main. Maybe something changed in the recent upgrades to grafana package in nixpkgs, not sure. As a temporary workaround you could use the nixpkgs pinned in test flake.