Observability and telemetry using grafana

[jennydaman]

Grafana is dashboarding software which can connect directly to PostgreSQL. It can visualize and answer questions such as:

• what is the most popular plugin?
• how many plugin instances are being created weekly?
• what is the failure rate for each plugin, and how is it changing over time?

Internal proof-of-concept: http://cube-next.tch.harvard.edu:9009/

The task here is to further explore grafana for the purposes of telemetry: what features of ChRIS are being used the most and what should we focus our development on?

[qxprakash]

@jennydaman POC link is not accessible .. ?

[rudolphpienaar]

This is only for the internal hospital CUBE. It is not publically accessible.

[sakshi1215]

@jennydaman @rudolphpienaar Just wanted to feed my opinions:

• Have we only connected PSQL to do the monitoring?
• Are we planning to instrument our code to get golden metrics or latencies?
• Have we done evaluation b/w self hosted or cloud-hosted instances?
• Have we finalised on the Observability tooling do we want to go with ELK(Elastic), PLG(Grafana+Prometheus+Loki), or something like Datadog which we can just configure without having worries of hosting/persistence/performance/cardinalities of the metrics?

[jennydaman]

@sakshi1215 we prefer to use self-hostable, open-source solutions if possible for the sake of data privacy. ChRIS targets deployments in healthcare settings so data privacy is crucial.

This issue needs research. It has not been explored in depth. So far, only Grafana+SQL has been tried. Please see my notes here: https://github.com/FNNDSC/miniChRIS-docker/wiki/Observability

I believe you have already joined our Matrix room? https://matrix.to/#/#chris-observability:fedora.im