Green-Software-Foundation
Prometheus Importer
Prize category
Best Plugin
Overview
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project's governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.
Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.
Problem/Solution Statement
We propose to build an Prometheus Importer which will facilitate the consumption and processing of metrics through Impact Framework, especially in a Kubernetes environment. We plan to allow PROM queries to be used as input for our model, with a specified aggregation window. The resulting array of measurements would be further processed by other models in the pipeline. This would be especially useful if and when the Impact Framework is deployed within a Kubernetes environment, significantly reducing the overhead of preparing the measurements to be used in the manifest file.
Scope and Limitations
- Prometheus is a prerequisite, either stand-alone or deployed via Kube-Prometheus-Stack
- the output array of the model will only contain the timestamp and metric fields and their respective values
Questions to be answered
- Are there any plans to containerize the Impact Framework?
- Are there any plans on exposing the Impact Framework via REST API? (especially useful if deployed as microservice in a Kubernetes Cluster to be used in conjunction with our model)
Have you got a project team yet?
Yes and we aren't recruiting
Project team
@ParvanAndrei @MXG99 @laurandrei994 @AmeliaCrizbasan @AlexHusleag @AngelCataron @hyper11011
Terms of Participation
- [X] I agree to the hackathon Rules & Terms and Code of Conduct
Project Submission
Summary
Our Prometheus Importer greatly facilitates the consumption and processing of metrics through Impact Framework, especially in a Kubernetes environment.
Problem
Apart from outputting standard metrics, custom PROM query can be used as input for our model, with a specified aggregation window. The resulting array of measurements would be further processed by other models in the pipeline. This would be especially useful if and when the Impact Framework is deployed within a Kubernetes environment, significantly reducing the overhead of preparing the measurements to be used in the manifest file.
Application
Our Impact Framework plug-in fetches automatically the most useful information regarding a system from Prometheus. The metrics exposed automatically are: CPU Utilization, Memory Available, Memory Used and Memory Utilization. Additionally, one can request an additional metric by using the 'Custom Query' input.
Prize category
Best Plugin
Judging criteria
Overall impact: Prometheus and especially the Kube-Prometheus-Stack is the de-facto monitoring solution within Kubernetes Clusters. Increased migration and development of new micro-services will only strengthen the choice of containers and container management systems, like Kubernetes.
Opportunity: Having enabled IF to rely on Prometheus as a data source in a pipeline, the solution can be used on an increasing number of operational environments, cloud infrastructures, cloud services, including container run-time services, etc.
Modular: Having developed this IF module as a data-source, it can be easily integrated with other modules in the downstream. It follows the convention in the output it delivers and it has been tested against both official and unofficial IF modules.
Video
https://youtu.be/f3sPHGQWGP0
Artefacts
https://github.com/andreic94/if-prometheus-importer
Usage
Process
In terms of development, we started from the IF model template project and then added the necessary implementations following the Azure Importer Module as reference.
Inspiration
While IF is powerful in it's capabilities, there are limitations in terms of tools/services it can use as input data. We are using Prometheus extensively, and we thought that giving IF this power will pave the road for further adoption by other individuals who are using this monitoring solution.
Challenges
Getting familiar with Typescript and its pitfalls reduced our efficiency in providing this module.
Accomplishments
We managed to develop everything we set out to do. The introduction of the 'custom query' input is definitely a highlight.
Learnings
Always get familiar with the language of choice beforehand.
What's next?
Having a way to asses the sustainability impact of micro-services, especially in containerized environments, will allow us to minimize the damage they cause to the eco-system.
@andreic94 We also intend to build a Prometheus importer plugin as part our hackathon. Can we find a way to collaborate together? cc @rossf7 @SFulpius
Hi @russelltrow ! We just saw in the DAY ONE email that there should be between 1 and 5 participants in a given team. Is this a HARD limit? If so, do you still accept additional issues to be created so that we can split the team? The splitting team members will need to go through the registration process once again for the new issue.
Thank you in advance for your reply!
Hey @andreic94, thanks for reaching out. We've discussed and decided to the relax the rule on maximum team size due to the level of interest around the hackathon.
You are free to continue with your existing team 😄
