bottlerocket icon indicating copy to clipboard operation
bottlerocket copied to clipboard

Published 20 hours ago • verified publisher icon bottlerocket-os

Out of the box "ECS Instance" Performance Monitoring for Container Insights

Open dfoley84 opened this issue 2 years ago • 2 comments

What I'd like: I would like to be able to have Cloudwatch Agent installed or pre-installed on an AWS Bottlerocket (ECS / EKS) AMI / Cloudwatch Agent running as a Daemon set within an ECS Cluster so that we can get instances-based metrics, into cloudwatch / Container Insights

Any alternatives you've considered: I have a FluentBit Daemon Set configured within my ECS Cluster, pushing logs to CloudWatch log group, but unsuccessfully, pooling for ECS Host Instance metrics. I've unsuccessfully tried to configure the cloud watch docker container as a daemon set within the ECS cluster.

dfoley84 avatar Sep 16 '22 21:09 dfoley84

@dfoley84, thanks for reaching out! At the moment we don't have plans to add logging agents directly to the host. There has been a little bit of discussion on this in bottlerocket-os/project-website#29, although it looks like you're already trying FluentBit. Can you provide us with a little more information regarding which metrics you're after?

jpculp avatar Sep 19 '22 20:09 jpculp

Hi @jpculp, I followed the Fluentbit blog, it appears that the input Syslog is not initializing on the amazon fluent bit docker container.

dfoley84 avatar Sep 20 '22 10:09 dfoley84

Hi @dfoley84, sorry for the delay. I think you'll want to try the Systemd input to get the logs from Journald.

jpculp avatar Oct 04 '22 00:10 jpculp

I'm looking in to this now. There have been at least a few users that have indicated having the CloudWatch agent in Bottlerocket would be useful. Currently scoping feasibility and seeing what would be involved with getting it added.

stmcginnis avatar Feb 14 '23 22:02 stmcginnis

Just adding a note - the CloudWatch agent should work well for performance metrics. I was hoping it would also work for Bottlerocket loggings, but it appears the CloudWatch Agent doesn't support reading from the journal yet, and all interesting Bottlerocket logs are logged there.

stmcginnis avatar Feb 15 '23 21:02 stmcginnis

Unassigning for now. After experimenting with CloudWatch metrics, it looks like there are quite a few that are captured by default without needing the CloudWatch agent installed on Bottlerocket. So while this may still be a good idea to add the agent, there maybe isn't as large of a gain as I had hoped.

Still worth others looking in to. From my experiments, this would entail:

  • Add new packages/amazon-cloudwatch-agent with Cargo.toml, amazon-cloudwatch-agent.json, amazon-cloudwatch-agent.service, and amazon-cloudwatch-agent.spec.
  • Configure amazon-cloudwatch-agent.spec to place all files in the right place on the filesystem
  • Configure amazon-cloudwatch-agent.service to start the agent, pointing to the path of amazon-cloudwatch-agent.json
  • Determine what settings would be needed under settings.cloudwatch, if any
  • Make amazon-cloudwatch-agent.json a template that gets populated by these settings
  • Add settings migrations to handle the new agent configuration settings

Likely more, but wanted to brain dump some of the things I'm aware of in case I, or someone else, comes back to this later so they have a few notes to follow. ;)

stmcginnis avatar Feb 21 '23 20:02 stmcginnis