Instrumentation and monitoring

[Envek]

Description

This PR demonstrates how to add instrumentation to Rails project with AnyCable using Yabeda and gives you local playground for monitoring.

It also includes a k6 example script to be used with xk6-cable plugin.

https://user-images.githubusercontent.com/1516722/132551690-81322d7a-74ae-448f-8ecd-e00e37660a06.mp4

How to use this example?

1. Start application with new prometheus and grafana components

   dip up -d grafana
   dip up rails anycable
   

2. Open Grafana at http://localhost:5000/ (login admin, password admin) and navigate to Dashboards → Manage.

   Or just open dashboards on direct links:

   • AnyCable (combined: both WS and RPC metrics): http://localhost:5000/d/anycable-combined/anycable-combined
   • AnyCable Go: http://localhost:5000/d/anycable-go/anycable-go
   • AnyCable RPC: http://localhost:5000/d/yabeda-anycable/anycable-rpc
   • Puma/Rails: http://localhost:5000/d/rails-puma/rails-puma
   • Docker containers CPU and RAM usage: http://localhost:5000/d/docker-containers/resource-consumption

3. Open demo application in other tabs or browsers, and look on beautiful graphs in Grafana!

Progress

• [x] Run Prometheus in Docker, setup metrics collection from other components (in production you will probably have it setup independently from your app)
• [x] Run Grafana in Docker, configure it with Prometheus and provisioned dashboards (in production you will probably have it setup independently from your app)
• [x] Enable AnyCable Go instrumentation
• [x] Instrument AnyCable RPC with yabeda-anycable gem and expose metrics with yabeda-prometheus
• [x] Instrument web server with yabeda-rails and yabeda-puma-plugin metrics
• [x] Get metrics about CPU and RAM usage (cadvisor)
• [x] Add dashboard for AnyCable RPC
• [x] Add dashboard for AnyCable Go
• [x] Add dashboard for Rails and Puma
• [x] Add dashboard for CPU and RAM

Technical implementation details

Materials used:

• Monitoring with Prometheus, Grafana & Docker Part 1: https://finestructure.co/blog/2016/5/16/monitoring-with-prometheus-grafana-docker-part-1
• How to automatically provision Grafana with dashboards: https://ops.tips/blog/initialize-grafana-with-preconfigured-dashboards/
• Grafana provisioning docs: https://grafana.com/docs/grafana/latest/administration/provisioning/
• Grafana docker image configuration: https://grafana.com/docs/grafana/latest/administration/configure-docker/
• Monitoring Docker container metrics using cAdvisor: https://prometheus.io/docs/guides/cadvisor/

Debugging:

• Check that Prometheus have all targets in “up” state at http://localhost:9090/targets
• Check that cAdvisor sees Docker containers of this demo app at http://localhost:8082/docker/