ECS FireLens compatibiltiy: Make it a drop in replacement for aws for fluent bit

[project0]

Is your feature request related to a problem? Please describe.

Most applications still send (structured/plain) logs to stdout and then depends on external scraping. Some opentelemetry sdk also do not support full OTLP log forward so there is no real alternative as of today.

Someone already using the opentelemetry collector for traces and metrics may also want to have a better integrated way with logs in ECS (Fargate). As of today there is no official documentation how to setup logging with the otel collector in AWS. Afaik there are only two native ways: awslogs (cloudwatch) or firelens.

I did some research and experimented how to setup otel collector and i am actually surprised it works pretty well, so i wonder why i did not find any resource :sweat_smile: As Firelens is at the end just fluentd forwarding to a sidecar container, i do not see any reasons why we cannot just use the otel collector instead of a additional fluentd sidecar.

Describe the solution you'd like

• Extend the components with the fluentforward receiver (native protocol support) and transform processor (required to parse json body logs)
• Provide example configs in docs

Additional context

I put the relevant configs and a example task definition together, maybe someone else wants to use it today, but the ADOT is not supporting the relevant components yet: https://github.com/project0/aws-ecs-firelens-opentelemetry

[mhausenblas]

@project0 thank you so much for this! Would you be willing to discuss this with me (ADOT PM) in some greater detail? I'm based out of Ireland, so shouldn't be too hard to find some time slot that works for both of us.

[luxaritas]

Also interested in this. @mhausenblas if you haven't already connected with @project0 and it would be helpful to provide any additional thoughts from my end, please let me know.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[luxaritas]

Keep open please!

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[jameshulse]

Let's bump this one more time - hoping there has been some movement on it?

[greaterthanmost]

Bumping this. This feature would be tremendously helpful!

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[luxaritas]

Bump to keep open

[project0]

Seems like @hyperdxio adopted this approach for ECS :-) : https://github.com/hyperdxio/aws-ecs-opentelemetry-sidecar/tree/main/examples