Nested Fields are output as "Logstash Config Format" Blob

[thejeff77]

I'm running into the same issue someone else ran into a while ago. The issue was closed because he didn't respond so I'll just open another one.

I have many fields that contain other fields and are nested to many levels. For example a field may contain the following:

"[{"Field_1"=>"Field_1_Value", "Field_2"=>"Field_2_Value", "Field_3"=>"Field_3_Value", "Field_4"=>"Field_4_Value", "Field_5"=>"Field_5_Value", "Parent_Field"=>[{"Child_Field_1"=>"Child_Field_1_Value", "Child_Field_2"=>nil, "Child_Field_3"=>"Child_Field_3_Value", "Child_Field_4"=>nil}]}]"

Is there a way to either have the plugin parse the contents as children of the parent field, or if there are existing filters that can parse the data properly?

I've tried all of the parse_method options. I assume they are "simple", "dig" and "flatten".

I can't use a json parser, as it isn't json.

The gsub option in the mutate plugin can make it into valid json, however I have to cut-and paste the code for every field name, and there are a lot of edge cases such as the BSON::ObjectId(), and various values without quotes around them so using gsub it isn't very maintainable or production-worthy.

It would be pretty great if nested fields would be parsed too, or at least supported by the dig parse_method or a new one.

the key value plugin runs on the whole set of data, and doesn't support custom delimiters like the '=>' used in this output format.

I'm using 0.4.1