containers-roadmap
containers-roadmap copied to clipboard
aws
[EKS] [request]: EKS Fargate Support for more Fluent Bit output plugins
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request I want to use logging to fluentd(forward output) or logstash(http output)
Which service(s) is this request for? EKS on Fargate
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Only CloudWatch, ES, Kinesis are available now as basic EKS Fargate fluent bit functions.
Are you currently working around this issue? I added a fluent-bit container in every pods.
Additional context We hope that the general http & forward output function will be implemented as soon as possible in order for EKS Fargate to become a more practical service.
Attachments
Thanks for raising this issue @Byeong-jun-Kim. Could you please expand a bit on what you'd like to see beyond what we described in Fluent Bit for Amazon EKS on AWS Fargate is here?
Thank you for your reply. The document says: 'You can choose between CloudWatch, ElasticSearch, Kinesis Firehose and Kinesis Streams as outputs.' I feel that the output support above is insufficient. Many Kubernetes users collect and process logs from fluent bit to fluentd, and send them to various places. Besides that, it would be great to have Forward Output which sent to the same stack, fluentd, and also http Output that can be used in general. Is my request a little clearer? @mhausenblas
@mhausenblas Adding another example here. In our case the HTTP output mentioned by @Byeong-jun-Kim would be great to have as we're using SumoLogic for our log system and this is the recommended way to send logs to SumoLogic. The Fluentbit documentation actually provides an example. Currently we have to go through a cumbersome configuration with Cloudwatch in-between. Thanks
To add to this I would like to see all native Fluentbit outputs as documented here: https://docs.fluentbit.io/manual/v/1.5/pipeline/outputs/. To start Datadog and Splunk would be my top two. As a work around you can use Kinesis Firehose, however that just adds cost and complexity.
+1
https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html Built-in EKS Fargate logging places some restrictions on the configurations you can set. The output plugins are validated and only these: firehose, kinesis_firehose, cloudwatch, cloudwatch_logs, and kinesis plugins are supported (others are rejected as the configmap is applied. We want the HTTP OUTPUT plugin to be enabled at least so we can provide another EKS pod to consume the same. Hopefully, this ticket will help prioritize the same!
Yeah, this limitation basically prevents us from using Fargate's logging functionality at all, until the GELF output is supported.
I'm curious, @Byeong-jun-Kim, how did adding a fluent-bit container to every pod work out for you? It seems to me like you wouldn't be able to have that pod access the "node"'s filesystem to read the logs. Or are you just have your pods log to a file on a e.g. emptyDir volume so that it can be read by the fluent-bit pod?
@philomory yes you're right We are processing the log with a fluent-bit by redirecting all the logs as a file. Since the storage space is limited, we designated a specific lifecycle by saving it by date. But it's very tricky and inefficient. That's why I made this request.
please, add support the kubernetes parser which is intended to enrich the logs with useful information like the namespace, container names, pod name and etc ... https://docs.fluentbit.io/manual/pipeline/filters/kubernetes
Would like to see the DataDog output supported as well. Currently can work around by Cloudwatch -> DataDog forwarder -> DataDog, but that's money I'd just rather not spend.
Greetings.
At our company, we use Coralogix for log storage and analysis.
Our logs get centralized in a Kafka cluster (confluent) and then consumed by an app that formats and forwards them to Coralogix.
The following config
[OUTPUT]
Name kafka
Match *
Brokers {{ .Values.logging.kafkaEndpoint }}:{{ .Values.logging.kafkaPort }}
Topics test-topic
Topic_Key topic
Format json
rdkafka.security.protocol SASL_SSL
rdkafka.sasl.mechanism PLAIN
rdkafka.sasl.username xxxx
rdkafka.sasl.password xxxx
rdkafka.queue.buffering.max.messages 500000
causes the following error
"0500-amazon-eks-fargate-configmaps-admission.amazonaws.com" denied the request: kafka is not a supported output plugin. Please fix the logging configmap
Please add support for Kafka outputs
We too would really appreciate the Datadog and Http plugins, is that on the roadmap?
~I'm looking forward to this issue but never participated on this forum before. Is there anywhere with the governance rules and possible timelines on how improvements are chosen? Is there anything I can do to help push this more?~
~Especially to Loki as a tail-end!~
Update: We moved everything out of Fargate into Immutable OS
@mhausenblas there's quite a reaction in this thread, would you be willing to take this issue further? Any kind of response will be appreciated, it's been over an year after you commented.
Oh dear, we are using ec2 node and considering switching to Fargate, however the current support of fluent-bit is limited and almost useless.
-
It doesn't support s3, so the log is not long term and cheap? Or we need to manually export from Cloudwatch to s3? That means we can only use Fargate as toy and not production ready?
-
For almost all of us have our existing log system, that means we need additional label or minimum mutation label on our log in a multiple environment settings.
-
Since everyone would likely need the Kubernetes metadata parser, wouldn't it be nice if default already parse? yet customer has minimum control?
-
What choice do we have now? Maybe i should plugin a fluentd side-car (500M binary) on my golang microservice (20M binary) just to support this limitation.
Hello,
One of my customer have below requirements as well. Kindly add this to the feature request as well:
Currently , Fargate provides a hook for logging with fluentbit: https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html .
Problem: AWS only exposes very specific plugins. https://github.com/aws/containers-roadmap/issues/1242 .
We need the fluent forward output plugin in order to get fluentbit fargate logs to Grafana Enterprise Logs (GEL).Fargate validates against the following supported output: es, firehose, kinesis_firehose, cloudwatch, cloudwatch_logs, and kinesis.
Request to AWS is to add “Fluent forward output plugin” to the above list.
Also, kindly advise if any workaround is there for the same.
Kudos to @Byeong-jun-Kim. Adding the "http" or "forward" output can tremendously improve the logging experience on EKS Fargate.
My organization has an existing ECS/lambda -> Fluentd on ECS -> on-premise Elasticsearch architecture for some reasons.
- Fluentd can congregate logs into batch requests before sending them to Elasticsearch, which greatly reduces network/CPU loading.
- We can enforce centralized data enrichment or manipulation rules. We can also do that by Elasticsearch ingest pipeline, but then we would lose the other advantages.
- Fluentd provides an extra layer of buffer between microservices and Elasticsearch so that in the unfortunate event of a network/Elasticsearch outage, the application logs are properly handled by Fluentd instead of drowning the microservices' resources.
- There is of course the cost-saving factor as we managed our on-premise Elasticsearch cluster.
All of these are not possible because the "forward" and "http" output filters are disabled. Now I change the code to log to both stdout and to a log file, mount that file to Fluent Bit sidecar, and add another sidecar to perform log rotation... It is a nightmare and is seriously hindering the adoption of AWS EKS on Fargate within my organization.
Please fix this, or at least let us know if "forward" and "http" output filter will be considered to be added.
Guys do you think will be possible to configure my promtail to scrap logs from fargate namespace?
