loki
loki copied to clipboard
grafana
Support for promtail to write logs to Kafka
Currently promtail supports reading logs from a Kafka topic, and write out to a Loki installation. The feature request is for promtail to have the ability to write logs to a Kafka cluster, the reason explained below.
Describe the solution you'd like Kafka can be found in many organisations, and is often times used as a way to have a central place for information to flow through. This sometimes includes logging information too. While promtail can read from a Kafka cluster and forward the logs to Loki, it still means that other tooling must be used to get those logs into Kafka. And when other tooling is used, perhaps the logs aren't written to Kafka in the best way for Loki (ie, missing labels, perhaps the logs are transformed from one format to another and so on).
Another reason for wanting to have Kafka in the middle ( log file > promtail > Kafka > promtail > Loki), is that it also opens up these logs for other use cases which Loki currently does not cater for (or ever will). For example, logs are now accessible for Machine Learning pipelines and real time analysis. Logs in Kafka could also be selectively exposed to external entities and so on.
By adding support for promtail to write to Kafka, this means that we capture the logs correctly from the get-go, in the correct format for Loki downstream (with all the right labelling).
Describe alternatives you've considered There are many other agents that can read log files and write to Kafka, but these don't necessarily do it in the best optimised way for Loki (ie, missing labels). It just creates a sub par experience once these logs are in Loki, as you're missing some critical metadata information to query and group on.
Can I pick this up? I have experience with Go, but not with Kafka, but am willing to dig deep. Can someone point to the exact files/changes I need to look at?
@wengelbrecht-grafana this is an interesting request. What would you think about making Loki a Kafka producer instead?
Hi @jeschkies Definitely an interesting proposition for Loki to write/produce to a Kafka cluster. I can certainly see how this could be usefull as well. I think it will come down to preference for customers to have Kafka before Loki, or Kafka after Loki. I'm not sure what the % preference split will look like in the community?
In the example I provided above (logfile > promtail > kafka > promtail > loki), we are trying to replace another solution where Kafka is used in front to feed logs to the other solution, and it would make sense in their architecture to slot loki also behind Kafka (and then promtail to manage the complete ingest pipeline before and after Kafka).
I guess both solutions have their advantages.
@deepto98 the one issue I see is configuration. Promtail's client configuration always assumes that the URL points to Loki. I'm not sure where a Kafka endpoint would fit in. The first action item would be to come up with a spec of the configuration.
I need this feature. if Promtail can support kafka, I can replace the fluent-bit which send the log for analysis.
same here, I would really like promtail to output to kafka, now using logstash to do the job. We like kafka in front of loki for multiple reasons; 1 - because other siem tools can also consume the messages out of kafka 2 - kafka will receive log messages while we can do maintenance on the promtail consumers or the loki back-end
because other siem tools
Out of curiosity, what SIEM tools are you using?
SIEM
loggie， https://github.com/loggie-io/loggie Fluent-bit， https://github.com/fluent/fluent-bit
@nature1995 not for now. However, you are more than welcome to make a proposal and implement it 😊
Hi, I'm quite interested in this as well, do we have any update on this so far?
Can I work on this issue I have good experience with both kafka and loki
@AlyHKafoury feel free to write a Loki Improvement Doc it's been a while that I thought about this. However, keep in mind that we will start recommending the Grafana Agent.
This would be a great feature., as it solves the problem with diverse networking requirements. We need this functionality to keep on using promtail as it only supports push to loki.
This would be a great feature. Can I work on this issue I have good experience with kafka
Hello, thanks for your feature request.
We're currently reevaluating promtails position as a project within Grafana Labs. Internally we're actually using the Agent for both metrics and logs collection at this point.
While we haven't made a formal decision yet, we expect in the near future that all new feature work will be done in the Agent's log collection pipelines rather than in Promtail.
n00b here, what happened? is it completed? is there a option in promtail that will let you forward/send logs to kafka? Any docs about this? Big thanks!
is there a option in promtail that will let you forward/send logs to kafka? Any docs about this?
No. The short version is that for the moment, and likely continuing going forward, we're not accepting new features for Promtail.
As I mentioned above, Grafana Agent has plenty of on-going feature work and a feature request there is the best recommendation I can give.