Persist Grafana Storage

[saraAlizadeh]

What is missing? I need to add a PV to grafana deployment. I found no sample in .jsonnet format. like the thing you've provided for prometheus here

Why do we need it? every time my grafana pod gets recreated, everything is missing. by everything, I mean custom dashboards, users, data sources, folders, permissions and so on . I need them to be persistent.

Environment

• kube-prometheus version:

0.4

Anything else we need to know?: my kubernetes version: 1.16

[brancz]

We intentionally setup Grafana in a stateless way, where all of Grafana is provisioned via configuration files managed via version control. That's the opinionated way how we choose to deploy Grafana and we don't intend to change that :)

[JoseArgueta]

Hi @brancz we have similar situation, is there any documentation to read about it?

[songrijie]

@saraAlizadeh We've implemented persist storage in our project in order to save accounts and privileges. Here is the way...

grafana-storage.libsonnet:

// In original volumes definition,  "grafana-storage" goes to emptydir.
// storageWithPVClaim replace "grafana-storage" in grafana deployment. 
local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet'; 

local pvc = k.core.v1.persistentVolumeClaim; 
local grafanaPvClaimName = 'grafana-storage';
local grafanaVolName = 'grafana-storage';
local vol = k.apps.v1beta1.deployment.mixin.spec.template.spec.volumesType;

// Convert to map for easier overloading, assumes all array elements are maps having "name" field
local toNamedMap(array) = { [x.name]: x for x in array };

// Convert back to array
local toNamedArray(map) = [{ name: x } + map[x] for x in std.objectFields(map)];

local grafanaStorageWithPVClaim(storageClassName) = {
    grafana+:: {
        deployment+: {
            spec+: {  
                template+: {
                    spec+: {
                        volumes: 
                        toNamedArray(toNamedMap(super.volumes) + toNamedMap([ 
                            vol.fromPersistentVolumeClaim(grafanaVolName, grafanaPvClaimName),
                        ]))
                    },
                },
            },
        },
        pvc: 
            pvc.new() +  
            pvc.mixin.spec.withAccessModes('ReadWriteOnce') +
            pvc.mixin.spec.resources.withRequests({ storage: '5Gi' }) +
            pvc.mixin.spec.withStorageClassName(storageClassName) +
            pvc.mixin.metadata.withNamespace($._config.namespace) + 
            pvc.mixin.metadata.withName(grafanaPvClaimName)
        
    },
};


{
  grafanaStorageWithPVClaim:: grafanaStorageWithPVClaim,
}

in entry jsonnet:

...
local grafanaPv = import 'wk-grafana-storage.libsonnet';

// default storageClass for monitoring.
local defaultPvcStorageName = 'wbox-log';

...
local kp = 
...
// grafana persistent storage
grafanaPv.grafanaStorageWithPVClaim(defaultPvcStorageName);
...

The solution introduces grafana-pvc.yaml, and it's up to you to keep it or not.

Not sure if there's better solution... hope it helps.

BTW, I agreed with @brancz that Grafana can go to stateless way, and we are moving to it soon as Grafana has connected to gitlab ce with oauth.

[TamasNeumer]

@brancz

Hi and thanks for the collaboration in the prometheus-operator!

I understand that the stateless approach would be the "best practice", however how should I proceed with alerts and notification channels then?

• Dashboards: The way I see I can export dashboards to configmaps, and these will be reloaded on pod restart.
• OAuth / OIDC config: The way I see it's also configured in a static way - pod restart won't effect it, users will be able to log in just as before.
• What happens if users create notification channels and create alerts for their dashboards? I guess these are gone with a pod restart, and won't be recreated once the new pod spins up. Is it possible to export these to ConfigMaps as well? Or would you rather suggest using the HA Prometheus-Alert for alerting and Grafana is merely a "visualization tool"?

Thanks!

[paulfantom]

Or would you rather suggest using the HA Prometheus-Alert for alerting and Grafana is merely a "visualization tool"?

Yes, this is our suggestion. Especially that kube-prometheus stack already deploys it.

[takeda]

Hi @brancz we have similar situation, is there any documentation to read about it?

Yes, this project looks great, but documentation absolutely sucks. It's here and there but you need to browse other repos and look source code. For people like me who never used jsonnet before it is a steep learning curve.

After looking I believe the intention is to configure dashboards using jsonnet: https://github.com/brancz/kubernetes-grafana#adding-dashboards

[patrickdung]

@songrijie Could you provide a more verbose example for this part

...
local kp = 
...
// grafana persistent storage
grafanaPv.grafanaStorageWithPVClaim(defaultPvcStorageName);
...

I have problem integrating this part into the entry jsonnet. I must have missed some other configuration / syntax required to utilize the grafana-storage.libjsonnet. Thanks.

[cateto]

@TamasNeumer hello thank you for your sharing.

Did you solve notification channels and alerts problem? Is it a critical problem?

[TamasNeumer]

@TamasNeumer hello thank you for your sharing.

Did you solve notification channels and alerts problem? Is it a critical problem?

Hi!

As far as I remember our Grafana was then set up with Postgres as a means for persistence. However we have strongly encouraged our users to use AlertManager + Prometheus for alerting.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.