Deeply parse JSON strings

[mtwilliams-code]

Sometimes events will have nested JSON that is stringify'd at different levels, resulting in events with structure like

  "name": "my-api",
  "serviceName": "my-api",
  "context": {
    "callbackWaitsForEmptyEventLoop": true,
    "functionVersion": "$LATEST",
    "functionName": "some-function-name",
    "memoryLimitInMB": "1024",
    "logGroupName": "some-function-name",
    "logStreamName": "2022/03/10/[$LATEST]8b6329fb30b54362a737fb013ffb3ee1",
    "invokedFunctionArn": "arn:aws:lambda:us-east-1:PHONE_NUMBER:function:some-function-name",
    "awsRequestId": "5556a991-4b86-4b8e-98a6-702f3871235f",
    "invocationCount": 1,
    "handlerType": "APIGateway",
    "authorizer": {},
    "correlationId": "0af3cf15-fceb-4402-ba0b-d2560fc3e3ad"
  },
  "hostname": "IP_ADDRESS",
  "pid": 9,
  "level": 30,
  "restResponse": "{\"statusCode\":200,\"headers\":{\"Access-Control-Allow-Headers\":\"X-Requested-With, X-Api-Client, X-Api-Client-version\",\"Access-Control-Allow-Methods\":\"*\",\"Access-Control-Allow-Origin\":\"*\"},\"body\":{\"tracked\":true}}",
  "msg": "Rest response info",
  "time": "2022-03-10T18:02:26.394Z",
  "src": {
    "file": "/var/task/src/handler/MyFunction.js",
    "line": 59,
    "func": "info"
  },
  "v": 0
}

The inner JSON string (in restResponse for this example) will not be decoded and split into fields in the document sent to elasticsearch.

🐞 Problem

These inner JSON strings should be decoded into their own fields as well, to more robustly allow for log pipelines to feed through to ES in a format that is searchable.

🎯 Goal

This would extend Kibana capabilities with captured logs.

💡 Possible Solutions

This could be resolved by attempting to JSON.parse any non-numerical field in logsToEd.js:buildSource. extractJson already checks first to see if the first character is {, so I imagine any performance impact from attempting to parse non-json strings would be minimal.