Service level metrics

[elephantum]

Implement technical metrics of webui, scheduler and executors like uptime, an average time to reread dagbag etc.

[smith-m]

It would probably make sense to outline what the minimum of these metrics should be.

Also what mechanisms are suggested to inject an http interface in process for scheduler and executor. In terms of providing an http component for metrics, this is probably in line with providing an http interface for a /health endpoint. (https://issues.apache.org/jira/browse/AIRFLOW-1084?jql=project%20%3D%20AIRFLOW%20AND%20text%20~%20health)

Also to note, there is one other prometheus plugin for airflow (https://github.com/airflow-plugins/pandora-plugin/blob/master/blueprints/metrics_blueprint.py) But to my knowledge, it also lacks this functionality

[f1yegor]

Some of the mentioned above metrics are available via statsd statistics of airflow itself. I'm using statsd-prometheus exporter to monitor them, the downside is that they are not prometheus-way of naming(e.g. airflow_dag_<dag name>_duration)