Support Input plugin too

[rverma-jm]

Can we support cloudwatch input for fluentbit. We already have something equivalent for fluentd. With the input plugin we can leverage the fluentbit sql streaming.

[PettitWesley]

@rverma-jm Since Fluent Bit only supports Go for outputs, this will have to wait till the C CloudWatch plugin is complete.

Can you provide more details on exactly what the input plugin would do and how you'd use it?

[hencrice]

@rverma-jm Could you elaborate on what would the CloudWatch input plugin for Fluent Bit do? Does it fetch logs or metrics from the CloudWatch service and forward to some where else? There might be better/more efficient ways to do that instead of using Fluent bit.

Since there's not much activity for more than 1 month. I will close it in a few days.

[PettitWesley]

@hencrice I think this should be labeled as a feature request, and kept open. I don't think this is something that we'll prioritize any time soon (may be never), but its worth keeping open so that others can comment and plus one.

I think he's thinking a Fluent Bit version of Fluentd's CW Logs input, which seems to just be able to fetch logs from a log group: https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs#in_cloudwatch_logs

I know this isn't strictly speaking a feature request on this repo, since this is the CW output, but I understand why @rverma-jm created it here, since the Fluentd CW plugin repo contains both the input and output plugin. We could move this to the AWS for Fluent Bit repo though.

[gchamon]

It would be nice to have a native polling plugin for CloudWatch logs, but at first glance, it seems like we could achieve the same with exec input to pull logs with aws-cli, rewrite tag filter to add the cloudwatch group to the tag and possibly a custom lua script to extract any relevant information from the logs before outputting

[muruguru]

I am interested in this input plugin to read cloudwatch logs using fluent-bit as well.

[soloradish]

We want move log entries from CloudWatch Log to AWS Managed OpenSearch service. Our pipeline look like EventBus --> CloudWatch --> OpenSearch . We deploy a ElasticAlert instance monitor events sent to eventbus continuously.

We could use filebeat to do that if we use vanilla ElasticSearch. ( Filebeat do not support Fine-grained enabled OpenSearch service).