multiple cloudwatch log groups per service ?

[sebastianovide]

Hello,

is it possible to define multiple Cloudwatch log groups per a single service?

the app we are containerizing with AWS Copilot writes multiple log files and we would like to send each of them to a different CloudWatch Log Group.

Any ideas?

Thanks

[sebastianovide]

I guess that installing AWS Cloudwatch Agent in the container may be a way ?

[KollaAdithya]

Hello @sebastianovide 👋

You can probably follow this approach

1. Instead of writing logs to multiple log files (stores on disk). You can configure you application/docker container to write logs to either Stdout or Stderr.
2. Configure a FireLensLogRouter a sidecar container using Copilot manifest https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#logging to deliver logs to multiple destinations.
3. You can also apply filtering on these logs from ECS container to send logs to different CloudWatch Log Groups.

Note: You need to attach additional IAM permissions using addons to the taskrole as mentioned here

Here are few examples of different use cases of FireLens on Amazon ECS Fargate. https://github.com/aws-samples/amazon-ecs-firelens-examples

Let us know if this works or you do not want to configure in this way!

[sebastianovide]

Hi @KollaAdithya,

I'm still having trouble understanding. For instance, if my app has customers.log and sells.log (we have about 5 files) and sends both logs to Stdout, how can FireLensLogRouter separate the Stdout to send the contents of the file customers.log to the log group customers and sells.log to the log group sells?

You can also apply filtering on these logs from the ECS container to send logs to different CloudWatch Log Groups.

This assumes that I can easily find some pattern to distinguish logs from the different files...

I was hoping for a simpler solution. In our current EC2 deployment, which we are trying to Dockerize using AWS Copilot, we have an AWS CloudWatch agent configured to handle sending all the files to different CloudWatch log groups.

I've managed to do the same with my Docker setup, but:

1. Is this the correct approach?
2. What is the best way to set permissions via IAM instead of saving the secrets locally?

[KollaAdithya]

Sorry for all the churn! After some investigation ECS Firelens also supports collecting from multiple log files to multiple destinations. You do not need to write to Stdout or Stderr.

Can you take look at this example on how you can implement it with ECS Firelens: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/ecs-log-collection#tutorial-2-same-as-1-but-run-on-fargate

You can also do it through CloudWatch agent. Can you take a look at this post on how to configure it

[sebastianovide]

thanks @KollaAdithya

From what I understood we can configure a sidecar container with Firelens. But I'm still confused about how can it read the logs files from the other container (the container running the app)

The only way I've found is described in https://aws.github.io/copilot-cli/docs/developing/sidecars/#efs-volume-in-both-the-service-and-sidecar-container . But from what I've understood it uses an ECS Volume (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/efs-volumes.html) while I would rather use a Docker volume for this (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-volumes.html)

Any way to do it with AWS Copilot ?

Thanks

update: in the mean time I did it installing cloudwatch agent in the main docker. It all works and it was pretty easy. It just doesn't feel right. I wish I could simply pass the list of logs files to the sidecar manifest and that AWS copilot would take of it automatically...

[iamhopaul123]

Hello @sebastianovide.

in the mean time I did it installing cloudwatch agent in the main docker. It all works and it was pretty easy. It just doesn't feel right.

I mean it's effectively a very similar solution as firelens. I don't think both solution can help you read the logs files from the other containers in real time tho. Either EFS or docker volume should be solutions for that.

I wish I could simply pass the list of logs files to the sidecar manifest and that AWS copilot would take of it automatically

Do you want to provide a list of logs files address and Copilot can automatically output these files to different log groups? For example

logging:
  destination:
    Name: cloudwatch
    region: us-west-2
    log_files: ["customers.log"]
    log_group_name: /copilot/foo-bar-customerlogs

[sebastianovide]

Do you want to provide a list of logs files address and Copilot can automatically output these files to different log groups? For example

yes, that would be great. To provide that config to the sidecar which can read the main container FS via docker volume. To abstract it to its simpler version, something like

logging:
  - log_files: "/app/logs/customers.log"
  - log_files: "/app/logs/sales.log"
  - log_files: "/app/logs/users.log"
  - log_files: "/var/logs/nginx.log"
  - log_files: "/var/logs/etc/some*.log"
  - log_files: ["one.log", "two.log"]
  - log_files: "three"
    log_stream: "yea.log"
  - log_files: "dog.log"
    log_stream: "yea-two.log"
    log_group: "/copilot/env-svc"