Automatic CloudWatch Intergration

[hossimo]

Describe the feature

It would be really handy if there was some global way to configure the CLI to append to cloudwatch log groups by configuration so that every command would attempt to write to a cloudwatch log group based on the profile used to run the command.

Adding a log option to the CLI conf file gives a seed as to where to send any logs, perhaps a log level to control the level of logs sent to the log group (info, warn, error, debug).

Perhaps this could work with other services (SNS, SQS, SES ?) but Cloudwatch seems like a sensible place to start.

Use Case

I start a S3 recursive cp ec2 instance to upload 1TB of data in a dethatched or screened terminal. the transfer partially fails:

2022-05-04T00:00:00-00:00  info - cp s3://<bucket>/<object> . --recursive
2022-05-04T00:00:00-00:00  info - s3://<bucket>/<object1> Started
2022-05-04T00:00:00-00:00  info - s3://<bucket>/<object2> Started
2022-05-04T00:00:00-00:00  info - s3://<bucket>/<object3> Started
2022-05-04T00:00:00-00:00  info - s3://<bucket>/<object4> Started
2022-05-04T01:51:00-00:00  info - s3://<bucket>/<object1> Copied 250GB @ 150MB/S
2022-05-04T00:30:00-00:00  error - s3://<bucket>/<object2> Failed [Error 5]
2022-05-04T01:51:00-00:00  info - s3://<bucket>/<object3> Copied 250GB @ 150MB/S
2022-05-04T01:55:00-00:00  error - s3://<bucket>/<object4> Failed, Ran out of disk space
2022-05-04T01:55:01-00:00  info - Job Completed, success: 2, failed: 2 Transfered 725GB

Proposed Solution

For example for a .aws/config

[default]
  region = ca-central-1
  output = json
  log = arn:aws:logs:ca-central-1:<account>:log-group:my-cli-logs:*
  log-level = info

Other Information

Obviously the logging would need to fail (gracefully/silently) so that failure to log does not fail the task by default, but maybe others will require that all CLI entries be logged.

There would also need to be some way to confirm that the CLI has permission is able to send logs to cloudwatch or other services.

All of this seems complicated to get right, but would greatly increase the utility of the CLI without having to require the user to make a shim just to implement logging.

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

CLI version used

1.18.147

Environment details (OS name and version, etc.)

All

[tim-finnigan]

Hi @hossimo thanks for the feature request. This does sound like it could be complicated as you suggested. Could you provide a little more context on what your use case would be?

Different services have various ways of configuring Cloudwatch logging so I’m not sure how a universal configuration would work.

If you just want to save every --debug log output for each command run then you maybe you could do something like create a workflow that automatically writes those log files to S3.

[hossimo]

Hey @tim-finnigan making a wrapper was my goto idea, however, I switch machines often and also do "custom" s3 transfers more often than not. I would write a wrapper that does what I need but I thought it might be a great option that allows you to log all services by a single configuration for "free".

I totally understand that this is not a simple addition as there would be a large number of edge cases. but for me, as I'm often working in S3 more often than not it's simple to write a one-off command to grab/send some files, but lately, I have been getting errors that were not being logged so I wasn't sure when they were happening.

In the short term, I'm going to write a wrapper that does exactly as you suggest, but it seems like a very helpful option to allow logging of commands with a small number of configurations and setup in cloud watch.

[DhwanishShah]

Hi, i want contribute in this project. Please assign this project to me @hossimo @tim-finnigan

[hossimo]

@DhwanishShah I can't make any changes to the assignment however I think you can commit a PR and mention this Issue.