[EKS][Fargate] [request]: Multi-line logging support with Fluent Bit

[SaloniSonpal]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request I want to have multi-line logging supported with the built-in Fluent Bit logging on EKS/Fargate.

Which service(s) is this request for? This could be Fargate, ECS, EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? A way to see multi-line logs in CloudWatch or other destinations.

Additional context I see this issue is open on the AWS for Fluent Bit distro side https://github.com/aws/aws-for-fluent-bit/issues/100

[PettitWesley]

Upstream has made progress on improved/generic multiline support: https://github.com/fluent/fluent-bit/issues/337#issuecomment-882953961

[zhonghui12]

Now we have aws-for-fluent-bit supports it from 2.18.0: https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.18.0.

[f0o]

Worth mentioning that this probably should also include multi-line logs split up by the container runtime due to size (related: https://github.com/aws/aws-for-fluent-bit/issues/25)

[liemng]

Is https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.18.0 available with EKS Fargate (Firelens) yet?

[PettitWesley]

@liemng you can use any Fluent Bit image with FireLens. FireLens is just a way to deploy Fluent Bit in ECS.

https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

[PettitWesley]

Also, for everyone, please note that we are tracking multiple bugs with the new multiline feature release here: https://github.com/aws/aws-for-fluent-bit/issues/100

So while you can use 2.18.0 with FireLens, the multiline feature doesn't actually work. I am working on investigating/root cause/fix. However, its looking like that will take a while. Follow the issue for updates.

@liemng

[PettitWesley]

Specifically, see my comments starting here: https://github.com/aws/aws-for-fluent-bit/issues/100#issuecomment-939577025

[selagnsknox]

is there any update when will this be available?

[rsumukha]

EKS Fargate has added support for builtin multiline parsers and it is enabled by default. https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#built-in-multiline-parsers

Configurable multiline parsers are not supported yet.

[PettitWesley]

@yixinz19 please elaborate, what issue are you facing, what is your config map.

[tri-bao]

this ticket was moved to "Just Shipped" on Jan 19. Does that mean it is available in current EKS on Fargate? Is there any configuration that I need to do to activate this feature? I followed this page to configure the configMap, it doesn't mention anything about multi-line logs https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html

[PettitWesley]

@tri-bao @hou-yimin EKS has rolled back this launch, so we have re-opened this issue and moved it to "Coming Soon". Apologies for the inconvenience here!!

[rnishtala-sumo]

@PettitWesley is there an ETA for this issue?

[sameeraksc]

Any update on this? we need configurable multiline feature in EKS

[DesAWSume]

Was there a progress on this please

[tri-bao]

hi @PettitWesley I saw this eks-charts release note https://github.com/aws/eks-charts/releases/tag/v0.0.152 mention aws-for-fluent-bit: support cri multiline parsing Does that means the feature is available? On which EKS version? Thanks.

[PettitWesley]

@tri-bao The EKS helm chart is different than the EKS Fargate built-in logging hidden container: https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html

The latest version of the helm chart supports multiline logging for all EKS EC2 versions.

[dev-travelex]

@PettitWesley : When will this INPUT option be available for EKS Fargate? As per this page,

The main sections included in a typical Fluent Conf are Service, Input, Filter, and Output. The Fargate log router however, only accepts:

The Filter and Output sections.

A Parser section.

If you provide any other sections, they will be rejected.

[florian-popchef]

@PettitWesley There is also a need on our side to be able to set a custom configuration for the INPUT section in EKS Fargate.

Any update on this?

[javierpdm8]

Hi Team,

This is a key feature for our implementation and I'm, really interested in knowing if there is any date when you are aiming to release this feature.

Thanks and regards.