AWS Metadata Filter Plugin for EC2/ECS Metadata

[PettitWesley]

I am considering building an 'AWS Metadata' plugin. It'd query the EC2 and ECS metadata services and add useful metadata to log records. For EC2, it'd use the new IMDSv2, since it is more secure if you're running applications on your instance that are exposed on the public internet.

Basically, replicate the functionality of these Fluentd plugins:

• https://github.com/takus/fluent-plugin-ec2-metadata
• https://github.com/michaelgruber/fluent-plugin-ecs-metadata-filter

EC2 metadata will be prioritized and implemented first.

Do you think this feature would be useful? Please plus one this issue.

Do you have requests for what it will support or how it will work? Please comment.

[vector-sec]

I have plans to use fluentbit to do a lot of log aggregation and I'd love to be able to decorate logs with information from EC2 metadata like account number, instance ID, region, IAM role, etc.

[nigels-com]

@PettitWesley I have some familiarity with the AWS-o-sphere. If you need/want help on this one, let me know.

[PettitWesley]

@nigels-com To be honest, I don't think I'll personally contribute much more for this plugin than the initial implementation in #1795. I need a simple "getting started with Fluent Bit dev" task to give to a co-worker, and I was thinking this would be ideal. We should start work on expanding it at the beginning of February.

That being said, if you or anyone else want this plugin expanded sooner, please go ahead and write the code for it- I can come up with a different task to use to train my coworker.

Let me also introduce myself a bit- I'm the creator of AWS for Fluent Bit, the AWS distribution of Fluent Bit. As well as FireLens, the ECS integration with Fluentd/Bit.

If you'd like to help out with my effort to better support AWS natively in Fluent Bit, consider reviewing the code in #1852- which is the first in a series of PRs to add standard AWS credential providers to Fluent Bit.

[PettitWesley]

Also... @nigels-com, are you one of the other "official" maintainers? Eduardo said that he recently added 2 folks.

[nigels-com]

Hello @PettitWesley I'm not an official maintainer, but I hang around here a bit.

[PettitWesley]

For EC2, it'd use the new IMDSv2, since it is more secure

Using V1 vs V2 of instance metadata will have to be a config option. V2 does not work for many users who will Fluent Bit in a container (i.e. most). I'm going to make V2 the default. If a user does not configure the plugin to use the correct version, it'll freeze Fluent Bit (because the request to retrieve metadata is synchronous/blocking).

[PettitWesley]

The next step for this will be to add support for ECS Metadata.

One request I have gotten is to support optionally adding docker labels.

[renanqts]

After https://aws.amazon.com/about-aws/whats-new/2022/01/instance-tags-amazon-ec2-instance-metadata-service/ Would be great to have this information inside the EC2 Metadata plugin

[PettitWesley]

@renanqts Agreed, its pretty easy to add new metadata fields if you want. See this for example: https://github.com/fluent/fluent-bit/pull/2605

[PrayagS]

@PettitWesley Any updates on this?

We are trying to set up Fluent Bit as a daemon service on ECS. Because of this, there is no Firelens and we lose the ability to decorate our logs with ECS metadata. Sure there are ways around it by modifying application code but we would prefer something global just like how Firelens does it.

[PettitWesley]

@PrayagS I just began working on the design for the filter this week itself. I can't promise any timeline and I don't know exactly when it will be out, but the goal/hope is to have a full ECS EC2 Daemon Support and Tutorial out for Fluent Bit by the end of the spring.

[PrayagS]

@PettitWesley That's great to hear. Looking forward to trying it out.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[PettitWesley]

I've mostly got it working: https://github.com/PettitWesley/fluent-bit/tree/filter-ecs

Hopefully I will get this released in one of the next 1.9 versions.

[mwarzynski]

PR which adds support for EC2 tags to the AWS Filter: https://github.com/fluent/fluent-bit/pull/6065

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.