scope icon indicating copy to clipboard operation
scope copied to clipboard

Replace Scope metrics timeseries with Prometheus metrics

Open bboreham opened this issue 5 years ago • 3 comments

Make the sparkline charts in the UI source their data from a Prometheus instance that has suitable labels*.

Also, if possible, make the "glass half full" overlay on nodes come from Prometheus too.

Benefit will be that probe can stop gathering this data, which is cpu-intensive, and stop uploading it, which is network-intensive. I expect it will make delta reports (#3677) much smaller.

By default Scope shows 15 seconds of data at 1-second resolution; generally Prometheus is run with 15-second or 1-minute resolution so the Scope charts would have to show a much bigger time window. I think this is a benefit, since the 15-second charts either dodge around too fast to gather much meaning, or are completely flat.

* probably "suitable labels" means this will be Kubernetes-only (at least as an MVP).

Related: #2664, #1619, #2832

bboreham avatar Oct 13 '19 14:10 bboreham

Note to have process-level metrics the target system would have to run something like https://github.com/ncabatoff/process-exporter

bboreham avatar Oct 14 '19 10:10 bboreham

@bboreham Does this issue assume the user of Scope will have Prometheus installed and accessible on their side or would we somehow install it ourselves with Scope? Also, if Prometheus wasn't there, should the metrics be absent or shall we use probe metrics as a fallback?

fbarl avatar Oct 15 '19 19:10 fbarl

I think we would make this optional, e.g. a command-line flag. So if the user knows they have Prometheus metrics available they can set the flag, and give a URL to reach Prometheus, and if not then Scope will collect and display metrics the way it does today.

bboreham avatar Feb 25 '20 11:02 bboreham