FLUME-3418 Add support for making metrics available to Prometheus

[tmgstevens]

Flume has a comprehensive metrics system build on MBeans which has exporters to HTTP/JSON and Ganglia.

Adding support for Prometheus, which is the de facto standard in the Kubernetes will open Flume up to a much wider range of deployment scenarios and similarly decouple Flume from the monitoring systems of corporates who now no longer ship Flume.

[rgoers]

So now Prometheus becomes a required dependency. Flume has always supported retrieving stats via HTTP. I've been doing that with Splunk for years. Would it be possible to do this in a flume-prometheus module so it can be optional?

[tmgstevens]

It's only turned on if you enable it via -Dflume.monitoring.type=prometheus. We don't really have a modular packaging system at the moment, which is a shame, but it does at least mean there's on JAR that does everything. The delta for this is there would be no impact for anybody using Ganglia or HTTP, but this adds in Prometheus/OpenMetrics format as well.

[rgoers]

Yes, I meant that Flume will now have required dependencies on

 <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient</artifactId>
    </dependency>

    <dependency>
      <groupId>io.prometheus</groupId>
      <artifactId>simpleclient_servlet</artifactId>
    </dependency>

even if the property isn't enabled. This is one reason I would prefer something like a flume-prometheus module so you only get those dependencies if you include flume-prometheus. But, as you say, we are packaging everything as a zip so putting it in a separate jar probably wouldn't buy anything right now. But hopefully it will in 2.0. Putting it in a separate module now might make that work easier.

[tmgstevens]

I've looked into refactoring the instrumentation package into modules. The challenge comes with https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoringType.java#L28 which would need to change to be a string rather than a class, and therefore we lose some strong typing.

I can change it, but I'm honestly not sold on the cost / benefit. @rgoers let me know which way you want me to go.

[rgoers]

Yes, I see the problem. No, you should not change it now. But this is exactly what I would like to change for 2.0. Flume does this is many places and it is too brittle. I would like to adopt a plugin system similar to what Log4j 2 does that allows new components like these to be added dynamically without having to update an enum file. i.e. - PrometheusHttpMetricsServer would be annotated to indicate it is a Plugin of type Metrics server with a name of PROMETHEUS (or the mixed case equivalent) so the loadMonitoring method in Application would be able to find it and start it without requiring an enum. This same stuff goes on for Sources, Sinks, Channels, etc. which makes user provided components second class citizens since they have to be configured via the fully qualified class name.

[tmgstevens]

@rgoers are you happy to merge this in please?