fluent-bit out_kafka_rest: Add support for Message_Key_Field in sending messages to Kafka Rest

Currently, Kafka Rest plugin supports only Message_Key which restricts in sending some predefined string as part of it. But this doesn't provide flexibility to choose one of the field from record to be considered as key while pushing the messages to Kafka Rest. Adding flexibility with choosing one of the field in record as Key (which internally is used for deciding partition in kafka) helps in a way that, all the co-related records are pushed to same partition.

Logic for populating Message Key in below mentioned order

If Message_Key_Field is defined and present in the record, use it
If Message_Key_Field is defined but not present in the record, look for Message_Key
If Message_Key is defined, use it
If Message_Key is also not defined, let Kafka choose the partition

Addresses # 7795

Testing Before we can approve your change; please submit the following in a comment:

[N/A ] Example configuration file for the change
[ N/A] Debug log output from testing the change
[ N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

[ Yes] Documentation required for this feature - https://github.com/fluent/fluent-bit-docs/pull/1175

Backporting

[Recommended] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Aug 09 '23 12:08 TechBRSavvy

@endersonmaia , @leonardo-albertovich , @fujimotos , @koleini

Can we have this reviewed?

Aug 12 '23 05:08 TechBRSavvy

@endersonmaia , @leonardo-albertovich , @fujimotos , @koleini

Ping, can we have this reviewed?

Aug 25 '23 05:08 TechBRSavvy

@endersonmaia , @leonardo-albertovich , @fujimotos , @koleini