cloud-on-k8s
cloud-on-k8s copied to clipboard
elastic
Logstash part of ECK
Hello World!
I see deploying Elasticsearch, Kibana and even APM, but NOT Logstash.
So there is my feature request)
We do not have any updates on Logstash as part of ECK at this time. Today, ECK can manage ingest node as part of a deployment that can be used alongside beats
Any insights on why logstash is not part of ECK? the ELK stack misses that important piece. There must be a really important reason to not put a priority on this. Can maintainers expose the reason for it?
Is there any update to this issue? The missing of logstash is the final blocker for me to deploy ECK
+1 on what is the product team reasoning for not including Logstash. Is there a better way to do import logs or some integration with Kube where we don't need it any more?
+. Even a year later, logstash support is not a priority?
The closest I could find was this medium article, will be attempting it shortly https://medium.com/@raphaeldelio/deploy-logstash-and-filebeat-on-kubernetes-with-eck-ssl-and-filebeat-d9f616737390
To all who are looking for the logstash feature set in ECK: Probably Ingest Pipelines is the missing feature. I realized this during the update to the latest version 7.8. Definition of pipelines is easy and can contain nearly all processors known from logstash. Just take a look in Kibana under Stack Management.
@wutkemtt , I took a look at ingest pipelines, but I don't see rabbitmq support in there like logstash does. Did I miss it?
Either way, using that medium article works, but I had to modify ES url to http instead of https.
The templates provided in that article require some quick modification to get it deployed as a deployment instead of a pod.
I just needed the processors, so I don't know about the RabbitMQ support, sorry.
this guide may help you to setup it https://github.com/framsouza/logstash-eck-ingress-controller
Probably Ingest Pipelines is the missing feature. I realized this during the update to the latest version 7.8. Definition of pipelines is easy and can contain nearly all processors known from logstash. Just take a look in Kibana under Stack Management.
Ingest pipelines are only able to receive data rather than consume it. This is a key difference from logstash which can, for instance, act as a consumer to pull events from a beats -> kafka -> logstash -> elasticsearch pipeline.
Some rationale from maintainers as to why logstash is not included in ECK CRDs/portfolio would be appreciated.
There is an official recipe for using Logstash with ECK: https://github.com/elastic/cloud-on-k8s/tree/main/config/recipes/logstash
There is nothing yet in the docs about this and it's obviously not a CRD for Logstash, but still may be helpful for some people.
I still don't get it why Logstash is not include. It has been 3 years already.
What's the point of deprecate the logstash chart if is not included in ECK?
I'm happy to announce that we are adding Logstash to ECK https://github.com/elastic/logstash/issues/15030
I'm happy to announce that we are adding Logstash to ECK elastic/logstash#15030
well, that's new... ;-)
thanks! after 3 long years))
Hey looks good!
How would you do this in the new operator model? https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html#installing-plugins
I would need to install the kafka plugin when using this.
This is support for a limited set up plugins built in, including kafka. We plan on adding more. You can see the use-facing documentation here - https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-logstash-technical-preview-limitations.html
