Ability to convert to Json ignored unknown values

[pentium10]

We love the plugin, and we primary use for event shipping.

We use the ignore_unknown_values config option but we would like to convert all ignored values to a json string something like convert_hash_to_json option does AND we would like to name to a column eg: meta

proposed config line:

convert_ignored_unknown_json_key meta #column name to hold the JSON string
time_string meta

this would take all ignored values and with their name would complile as JSON and place in the column named meta

expected in meta (in Bigquery there is no column for host/ip/port/user:

"meta" => "{\"host\":\"remote.example\",\"ip\":\"192.0.2.1\",\"port\":12345,\"user\":\"tagomoris\"}"

for us it's really high priority, let me know once you read it, what you think.

[pentium10]

@joker1007 what you say about this?

[joker1007]

I think that this feature is substitutable by other filter plugin. For example, fluent-plugin-record-transformer or fluent-plugin-record-reformer or fluent-plugin-record-modifier. If other plugin can realize your demand, it is better to use the other plugin. Because record transforming is not main purpose of this plugin. How about this?

[pentium10]

The problem is that if use another plugin we need to duplicate the field definition on multiple places. Practically I need to hardcode all the fields that exists so a plugin can detect what's missing. And that needs constant maintenance. It was extremely handful and great that here with auto table schema we should not describe the fields, only some the meta field. On top of the above reasons, it's a very known feature of hybrid columnar storage system to have this functionality in place, place rest of the fields as json in a data/meta column. Does this change your thinking? Is this something complex to achieve?

[joker1007]

I see your probrem. Please give me time to think about it.