Prometheus exporter

[edomora97]

Add a new command: cmsPrometheusExporter

This will start a new HTTP server that listens on a port (defaults to 8811) for requests to /metrics. At this endpoint many metrics about a contest (or all contests) are exposed for Prometheus to be collected. Those metrics consider both system values (queue status, connected workers, ...) and contest values (submissions, users, questions, ...)

This exporter works by executing simple queries to the database, and obtains the queue information asking Evaluation Service.

The metrics exposed may leak information about the contest, so it's important to secure this endpoint (for example using a reverse proxy, or binding localhost).

With this exporter it's possible to build interactive and real-time dashboard for monitoring the status of the contest. For example, you can find here the dashboard we used at OII 2020, and here the one used at an OIS round. Those dashboards are built using Grafana, and we configured it also for sending telegram notifications when something wrong was happening (e.g. queue too long, see image below).