Add Universal Collector to collect metrics from files or HTTP endpoint

[vgratian]

Is your feature request related to a problem? Please describe. Harvest 1.6 supported running extensions (scripts in any language) to easily collect custom metrics. One example is collecting NFS mount information from ONTAP CLI. But extensions were able to collect any metric from any source. This functionality is missing in Harvest 2.0. Many users already have some scripts to collect custom metrics which they can not easily integrate to Harvest. Currently only solution is to rewrite those scripts in Go an create a Harvest collector.

Describe the solution you'd like Idea was suggested by @georgmey. Create a collector that will collect metrics in open-metric-format from files or a remote HTTP endpoint. This would be very similar to Prometheus' node exporter.

Describe alternatives you've considered Implementing the same Extension framework that we had in Harvest 1.6 would require too much work. Moreover, running custom scripts as part of Harvest runtime would raise considerable security concerns (one of the reasons we discontinued 1.6).

Additional context The general workflow of the Universal collector:

• User has a custom script or program that generates metrics and writes them either in a file on exposes on an HTTP endpoint
• Universal Collector collects those metrics and parses them into the Matrix.
• User can choose to which databases to export these metrics (as per usual).

The Universal collector will be only good for small number of metrics. It would not be a good solution for large chunks of metrics where high-performance is critical.

[cgrinds]

Seems like this might be a better fit for Telegraf.

[cgrinds]

Covered by existing plugins