json adapter

[progrium]

This is a generic JSON adapter that can be used with Logstash, but doesn't need to be. Its fields can be configured with environment variables, with minimal default fields enabled, and a few useful premade fields that can be enabled.

[NoumanSaleem]

@progrium any update on this?

[tumd]

I would really love to see this feature merged. Would do it myself, but not sure how to resolve the conflicts.

[geoffreytran]

Another +1 on getting this merged.

[dkiser]

@progrium Any way we can get map types properly converted to json before this PR is merged? I tried running this branch on a dev box to see what {{.Container.Config.Labels}} would look like, but it doesn't appear the types map properly reads maps to end up being json marshalled correctly.

example json pretty print of a syslog message with json adapter and labels field

{
  "docker": {
    "hostname": "6965fdb0f180",
    "image": "centos:centos7",
    "labels": "map[name:CentOS Base Image vendor:CentOS build-date:2015-12-23 license:GPLv2]"
  },
  "message": "test",
  "time": "1453494436"
}

Ran with following env vars

JSON_FIELDS="time:uint,message,docker.hostname,docker.image,docker.labels"
JSON_DOCKER_LABELS="{{.Container.Config.Labels}}"

Thoughts?

[progrium]

That looks like an easy fix. Also I'm surprised this was never merged. Maybe it would be a good idea to merge this and then talk about fixing that as a separate issue that others might be able to help with?

[dkiser]

@progrium either way is fine.

Any ideas how the marshalling would be fixed? Looking at golang json docs seems like it would be marshalled correctly by default for known types. Is the use of the template render causing this? Haven't looked too deeply at it plus I'm a go beginner.

[progrium]

Yeah I'm assuming it's some oversight where I assume some variable is going to be a string. I haven't looked at the code in a while. I'm just, you know, swamped. Also still looking for another maintainer for this project.

[progrium]

Ugh, and it needs to be rebased now too. Maybe neeeext week? :\

[dkiser]

Hehe No rush on my part. I can always build and use locally with the changes 8-).

[dkiser]

@progrium one thing I noticed was this code is using the stock udp transport. I had a specific need to send JSON over a valid syslog message. So i'll possibly fork and re-use some of this json code by adding a new MODE to the syslog adaptor configured by environment variables. The idea would be to send JSON payload in the "message" portion of a syslog message where this payload is configured via environment variables as you've already started on this branch.

High level goal is JSON payload + syslog (udp or tcp).

[progrium]

No reason adapters can't use each other. If there was a way to refactor this so that syslog could be configured to use it (and all its JSON configuration) in its messages that would be rad. Try and explore that in your fork and if you come up with anything let us know!