bunyanToGelf function needs additional argument to control flattening

[jcferrer]

Adding flattened properties as additional fields in the GELF structure, is certainly a very nice feature which allows this properties to be "searchable" in the Graylog GUI. Basically all of this fields are indexed by the ElasticSearch engine.

However this has a very adverse effect when the actual full_message contains JSON payloads that contain req/res type properties, like those captured to log roundtrip http request.

For instance:

"res": {
    "body": {
      "status":400,
.....
}

adds this GELF extra field:

"res.body.status":400,

Everything is great until, we try to log another response with:

"res": {
    "body": {
      "status":"ACTIVE",
.....
}

"res.body.status":"ACTIVE",

Then graylog server rejects the message due to a "MapperParsingException", Basically because of the indexing criteria, assuming in this case that res.body.status should be numeric. Basically the first type that made it through the log first, wins.

The solution is not to automatically add fields after flattening the log. So maybe bunyanToGelf can take an extra argument to selectively pick this behavior.

bunyanToGelf(log, addFlattened) {
......

  if (addFlattened) {
    for (key in flattenedLog) {
      if (ignoreFields.indexOf(key) < 0 && gelfMsg[key] == null)
        gelfMsg[key] = flattenedLog[key]
    }
  }

....
}

[mhart]

Oh wow, that's kinda lame. I wonder if that's new? (I haven't used Graylog in a few years)

Is that honestly the only solution? There's no way to make the types dynamic on the server?

[jcferrer]

It is actually a documented problem. The issue is with the way ElasticSearch handles the indexing.

[jcferrer]

So do you think you can address this?

[mhart]

Well you can use your own mapping function:

function myBunyanToGelf(log) {
  // put your own flattening behaviour here...
}
var stream = gelfStream.create('localhost', {map: myBunyanToGelf})

Not sure whether it makes sense to add the flattening behaviour as an option considering it's just supposed to be a little helper function (and it's trivial to add your own)

[jcferrer]

Ok, I guess I didn't know I could do that!

Thanks

[mhart]

Cool, no probs – also all of the functions are exported, so you can use them in your mapping function too if you like (eg, gelfStream.mapGelfLevel, etc)