kafkaexporter: Allowed for a keyed producer

[MovieStoreGuy]

Is your feature request related to a problem? Please describe. The idea being that there is a way that data can be keyed when sending to Kafka to allow for ordered messages. It could be potentially useful who use an ordered dependant system and can not easily reconcile out of order data.

Describe the solution you'd like The ability to set the key data field(s) within the kafka exporter config in order to send keyed messages.

Describe alternatives you've considered There is ability to set a static keyed method (ie. keyed: "time", keyed:"resourceName") however, I am happy to further conversation here.

[rcrowe]

Noticed https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/5264 was raised by @Frefreak but went stale.

Does that PR satisfy the needs of this issue & worth revising? If not, what would the implementation of a keyed producer look like so that others can contribute?

[Frefreak]

Here a configurable "properties" to be keyed is wanted, but my PR only keyed the message by traceID though (intended to be used with tail-based sampling). Maybe that PR can be generalized (or make a new one) to support a pre-defined list/enum of attributes to be keyed.

[MovieStoreGuy]

I am to start simple and make it more complex as we need :)

I was thinking that a user could configure a keyby by using some combination of name, time, resources, libraryname and perhaps attributes so you can do something like:

keyBy:
- field: time
  round: second
- field: name
- field: resources
- field: trace
  default: unknown

[xiangtianyu]

Is there a sequel to this?

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[rydevops]

I am not sure if this can be achieved but if we did the keyBy option and allow us to use things like the traceId, spanId and/or the service.name, that would be fantastic for helping to ensure correct ordering. If this feature is added I can always do some testing if required.

@MovieStoreGuy anyway to get this out of stale?

[fredthomsen]

My thoughts here were to add an OTTL expression as part of the component config in order to point at the proper field to key off of. I have not seen any OTTL use in exporters and am not aware of all the limitations. Does this idea seem reasonable?

[github-actions[bot]]

Pinging code owners for exporter/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MovieStoreGuy]

I am not sure if this can be achieved but if we did the keyBy option and allow us to use things like the traceId, spanId and/or the service.name, that would be fantastic for helping to ensure correct ordering. If this feature is added I can always do some testing if required.

@MovieStoreGuy anyway to get this out of stale?

Sorted :)

[MovieStoreGuy]

My thoughts here were to add an OTTL expression as part of the component config in order to point at the proper field to key off of. I have not seen any OTTL use in exporters and am not aware of all the limitations. Does this idea seem reasonable?

I really like this idea, reducing the amount of friction and improving the reusability is something I care deeply about. If anyone wanted to start on this, I would be more than happy to review.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SHaaD94]

Hi guys. Is anyone working on this? We really need this feature :)

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.