Manage external grafana instances with the operator

[hubeadmin]

Is your feature request related to a problem? Please describe. The idea is to allow the operator to be used partially i.e. only dashboard/datasource/etc... reconcilers on an external (non-operator managed) instance of Grafana, this would allow users to maintain a central Grafana instance for multi-cluster monitoring.

(If applicable)If your feature request solves a bug please provide a link to the community issue

Describe the solution you'd like The ability to define a new externalGrafana struct with necessary credentials/envFroms/secrets etc. and have the operator make API requests against a non-operator managed Grafana instance to CRUD dashboards/datasources (or whichever other resources will be supported in the future by the operator).

Describe alternatives you've considered Not applicable, the idea is to have the operator manage an instance outside of the cluster in which it is running, so it's not currently possible with the operator

Additional context Thread that kicked off this proposal: https://kubernetes.slack.com/archives/C019A1KTYKC/p1619417347016600

Existing solutions Similar solution is planned for Loki & Promtail

[cheddarwhizzy]

I'm also interested in making this a reality.

These would be the only keys necessary since the rest of them are for configuring the managed Grafana deployment from what I can tell.

spec:
  client:
    external:
      host: https://some.remote.grafana.domain.com
      user: <api-user>
      pass: <pw>
      additional_headers: {}
 dashboardLabelSelectors: {}
 deployment:
    enabled: false # auto-disable deployment if spec.client.external is set

[pb82]

There are a few considerations.

• We won't be able to manage plugins of an external instance.
• It requires some refactoring of the way the operator manages the credentials.
• We won't be able to import data sources to the external instance (unless we also rewrite that to use the API).

We're still open to a contribution if anyone wants to take this on and can discuss the details here or on Slack.

[NissesSenap]

Considering scope creep with supporting external Grafana instances https://github.com/grafana-operator/grafana-operator/issues/497 how interesting is this still?

Do we think this still is a reasonable feature to add to the operator?

[cheddarwhizzy]

While this would be a good feature to have, I think we're going to use kubefed to sync our GrafanaDashboard resources between our source clusters and our destination cluster running the Grafana operator, and set the replicas to 0 for the operator and the Grafana deployment in the source clusters (or only install the CRDs). This way we don't lose the plugin management capabilities, and no need to touch source code :)

[NissesSenap]

@cheddarwhizzy that sounds really interesting, if you manage to get it to work it would be really nice to have a blog post or something similar on how you solved it.

[elamaran11]

@cheddarwhizzy Please let us know if we have a working solution for this. I have a need to manage remote Grafana Instance with Amazon Managed Grafana to create Dashboards.

[hubeadmin]

Resolved in #919