feat[grafana]: scripting dashboards

[jeqo]

trafficstars

Major change to manage grafana dashboards via scripts, using grafanalib.

Changes:

• New directory for dashboard scripts: ./grafana-dashboards

• Script to generate dashboards for VM/cp-demo deployments and CFK deployments: ./grafana-dashboards/Makefile

• Updated dashboards:

  • jmxexporter-prometheus-grafana/assets/grafana/provisioning/dashboards
  • cfk-prometheus-grafana/grafana

• Dashboards:

  • Kafka Cluster:
    • Remove "Broker network throughput" panel, and use it for specific metrics

- Move Zookeeper related metrics to Zookeeper cluster dashboard 
- Throughput panels that sum all brokers are removed, as that can be inferred from throughput per brokers, or tweaked by stacking the results per broker: 

- Request rates put together in one timeseries:

• Kafka Connect cluster:

  • Redesign the general group, removing pie-charts and showing table with all details instead, also moved rebalancing up in the dashboard:

  • Mixed related metrics in the same panel: e.g. commit success and failure % are now in the same panel

• Kafka Topics:

  • Removed the topic/partition stat as it is on the kafka cluster
  • Merged offset tables into 1

• ksqlDB:

  • Reduced stats on the top to show more in less space:

[waliaabhishek]

Kafka Consumer dashboard -- Added 32 new panels. Awesome work here. Original had 19 expr statements. New has 51 expr statements.

Kafka Producer dashboard -- Added 10 new panels. Awesome work here. Original had 32 expr statements. New has 42 expr statements.

We are missing some graphs on the newly generated json files:

Kafka Cluster dashboard -- Missing 16 panels Original had 70 expr statements. New has 54 expr statements.

Kafka Connect dashboard -- Missing 22 panels Original had 74 expr statements. New has 52 expr statements.

Kafka Topics dashboard -- Missing 2 panels Original had 10 expr statements. New has 8 expr statements.

KSQLDB dashboard -- Missing 5 panels Original had 39 expr statements. New has 34 expr statements.

SR dashboard -- Missing 4 panels Original had 12 expr statements. New has 8 expr statements.

[waliaabhishek]

The CFK new commit looks awesome. Thanks for the new add.

[jeqo]

@waliaabhishek thanks for the review! I have added panel changes to the description with some of the reasoning behind removing/merging/moving some panels, also fix a few based on your feedback. Let me know how it looks now.

[cla-assistant[bot]]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}