ecs-watchbot icon indicating copy to clipboard operation
ecs-watchbot copied to clipboard

Add support for nested structured logging

Open danpat opened this issue 3 years ago • 0 comments

Over in https://github.com/mapbox/vorpal-sword we've got a worker that is capable of emitting JSON structured logging, which CloudWatch understands how to parse, and make nice-looking queries based on certain fields in log messages.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html

Currently, the watchbot worker logger adds a string prefix to any stderr/stdout output from the worker process, which breaks the JSON format, and causes CloudWatch not to recognize the log lines as JSON formatted. Example:

[Sat, 30 Oct 2021 02:17:04  GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399]  {"name":"@mapbox/vorpal-sword/backend","watchbot_session_id":"45e24f54-86d4-4952-8666-8146042985d6","hostname":"7dc1b301967c","pid":4334,"sns_message_id":"0f29eaca-18f3-5230-b8d0-b33c323c133e","sqs_message_id":"58598cd1-a09e-4d16-be9a-2355b86c1399","problem_id":"7dccbd5c-15f3-4280-908b-5e7c1917acaa","originating_request_id":"JWTRm-jMTDaWPKhc7mZBKOgkB8b4Ad9LV89UNwDJKhzv-iIEO19SUA==","level":20,"timestamp":"2021-10-30T02:17:04.432+00:00","logger":"stderr","process_id":4353,"thread_id":4353,"msg":"Adding  service 186 at location 186 as node  188","event":"worker_vorpal_sword_log_line","time":"2021-10-30T02:17:04.448Z","v":0}

If the string prefix of [Sat, 30 Oct 2021 02:17:04 GMT] [worker] [58598cd1-a09e-4d16-be9a-2355b86c1399] was not there, then it would be possible to perform cloudwatch log insights queries like:

fields @timestamp, @message
| filter originating_request_id like "JWTRm-jMTDaWPKhc7mZBKOgkB8b4Ad9LV89UNwDJKhzv-iIEO19SUA=="

To trivially find all messages that correlate with an original CloudFront request ID. Of course the fields here are specific to our system.

There are two ways we could address this:

  1. Add a config flag to ecs-watchbot like emitWorkerLogLeader: false to just turn it off. That would put the onus on the worker process to emit the relevant fields (like timestamp, watchbot session ID, etc).
  2. Add a config flag like structuredLogging: true - in this case, watchbot would expect the client to emit JSON formatted messages, and would merge watchbot specific fields into the JSON.

danpat avatar Oct 30 '21 16:10 danpat