Feature Request: Move from Monitoring Platform to Observability Platform

[ricsanfre]

Feature Request

Improve current Monitoring platform, based on Prometheus (metrics) and EFK (logs) to a Observability Platform adding traces monitoring and a single plane of glass.

Explore solution based on Loki (logs), Tempo (traces), Prometheus (metrics) and Grafana as single plane of glass for monitoring.

Main requirements

• Loki as complement of the current EFK platform, not a replacement. ES is used mainly for Log Analytics (log content is completely indexed) while Loki can be used for Observability (only log labels are indexed) having together logs, metrics and traces in the same Grafana Dashboards.

• Current logs collection/distrution layer based on fluentbit/fluentd should be used to feed logs to ES and to Loki, instead of deploying a separate collector (Loki promtail)

  Fluentbit/Fluentd Loki plugins are available, so current forwarder/aggregator architecture could be maintained just adding Loki as backend for logs.

  • Fluentbit Loki output plugins
  • Fluentd Loki output plugin plugin maintained by Grafana community

• Linkerd distributed tracing mechanism should be enabled. See this Grafana's blog entry where the process is explained.