allow to have a dedicated key for headers content

[ttych]

trafficstars

Hi Maintainers !

It is a pull request, to extend a current feature.

It is possible for some inputs to inject kafka message headers into the message (merge). But for some scenario, there can be some overlap between kafka message keys and headers keys, and headers overwrite message content during merge with headers content.

Even the add_headers was not present in the basic kafka input.

So the purpose of this PR is to :

• have the add_headers feature available in every inputs
• add another variable to control the header merge into record
  • if headers_key is set, the headers content will be pushed into the record under this key

I tried to add some tests, while tests are working fine (with my local kafka instance) with test/plugin/test_in_kafka.rb, tests are flappy with test/plugin/test_in_kafka_group.rb (I don't know if it is related to the kafka settings on my desktop or related to the kafka input processing).

Thanks,

Regards,

[ttych]

Hi,

Don't hesitate to comment, amend, reject the PR ;) My motivation behind this is to be able to split and identify what comes from the message "body" and what comes from the message "header". Thanks.

[github-actions[bot]]

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days