Add multi-namespace support for GrafanaDataSource

[jsanda]

Is your feature request related to a problem? Please describe. I am using grafana-operator in the k8ssandra project. In short the project provides Helm charts for deploying Cassandra clusters with supporting tooling for monitoring, backup/restore, etc. There are scenarios in which the user would want to use a single Grafana instance to monitor multiple Cassandra clusters, which presumably are deployed in different namespaces.

(If applicable)If your feature request solves a bug please provide a link to the community issue https://github.com/integr8ly/grafana-operator/issues/303

Describe the solution you'd like I want to be able to deploy a single instance of the operator and have it manage GrafanaDatasources across multiple namespaces.

[david-martin]

This use case makes sense. A similar model to how GrafanaDashboards are handled could be adopted e.g. have a dataSourceSelector & dataSourceNamespaceSelector in the Grafana CR. I don't believe there's anything blocking this.

However, it would be good to refactor the datasource management logic to use the API rather than configmap (and restarting grafana).

[drewwells]

Is this still an issue? I see a flag for watching all namespaces. I don't see why this flag is required namespace == "" should be sufficient for watching all namespaces. https://github.com/integr8ly/grafana-operator/blob/master/cmd/manager/main.go#L63

[AdheipSingh]

I see a flag for watching all namespaces. I don't see why this flag is required namespace == "" should be sufficient for watching all namespaces.

How about having a deny_list ? watch all namespace NOT kube-system, abc etc . We can use predicates to do that, should be quick and clean. Lets deny the ns events before passing it on workerqueue cc @pb82 @david-martin @HubertStefanski

[drewwells]

Controller runtime will handle those optimizations for you. You don’t need to watch all events for each namespace. It applies a predicate on the watch that limits to only those resources you are interested in for any namespaces present or future that contains resources. The reconciler is presented with a full view of all objects at that version of state in the cluster.

https://pkg.go.dev/sigs.k8s.io/controller-runtime#hdr-Clients_and_Caches

[AdheipSingh]

yeah we still need to implement predicates for that :)

[hubeadmin]

Is this still an issue? I see a flag for watching all namespaces. I don't see why this flag is required namespace == "" should be sufficient for watching all namespaces. https://github.com/integr8ly/grafana-operator/blob/master/cmd/manager/main.go#L63

I believe that the flag is required by some of our other teams due to the way installations are handled by their observability stacks.

to implement multi-namespace datasources, we'd like to refactor them to more closely resemble dashboard logic, i.e selectors etc. as @david-martin pointed out, the logic is already there, it just needs to be replicated to work for datasources.

as for deny_listing system namespaces for discovery, I think that's a good idea, any thoughts on this @pb82 ?

[pb82]

@HubertStefanski deny_list can make sense in the current model we have where the dashboard (and datasource) selection happens in the Grafana CR. If we flip this model (selector for Grafana instance in the Dashboard CR), then this would be less (but still somewhat) useful.

In general a deny list can be added at a later point too as there are always ways to work around it.

[hubeadmin]

Resolved in #919