amazon-cloudwatch-logs-for-fluent-bit icon indicating copy to clipboard operation
amazon-cloudwatch-logs-for-fluent-bit copied to clipboard

Feature request: field indexes config for CW log groups

Open strowk opened this issue 7 months ago • 0 comments

The feature of field indexes was announced several months ago - https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-cloudwatch-logs-field-indexes-log-group-selection-log-insights/

It would be good idea to support this configuration for log groups managed by cloudwatch container insights agent (that is AFAIK uses fluentbit under the hood) somewhere around this place - https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/blob/7b8dae0d6dedf942bc1e9feded197b8d1a9d261a/cloudwatch/cloudwatch.go#L600-L619

Such addition would allow us to index fields that container insights injects to log messages, such as kubernetes.container_name and so on. This should significantly speed up and reduce scan size (which I assume translates to scanning costs) for most of actual use scenarios when only particular service in question is being investigated (or queried by alerting systems). Without this it would seem to me that cloudwatch is not suitable for actual production usage as a convenient log system and we were considering already where to switch for awhile. Albeit not complete set that we would need, but it can be a good start.

strowk avatar Apr 04 '25 12:04 strowk