zeebe-kafka-exporter
zeebe-kafka-exporter copied to clipboard
camunda-community-hub
Enhancement: custom header/metadata
Hello! It would be useful to be able to include additional, custom information with the messages that are published to Kafka.
The use case that I'm thinking of is running multiple Zeebe clusters (e.g. during a migration since horizontal scaling is not supported). Assuming, the same Kafka topics are reused, it would be useful to know which Zeebe cluster produced each event.
I was originally thinking that this could be supported by providing static attributes that would be merged into the JSON payload for each message.
But with Kafka support for headers, maybe this is better supported by specifying attributes to be included as headers in the exporter configuration.
What do you think?
Sounds reasonable, but keep in mind that the information we have available is typically just the record - if we wanted more info available to the exporter, then we should open an issue on the Zeebe repo to extend the exporter API.
But I think your proposal is a good first step, if I understand correctly - we add a new config property where users can add static attributes (e.g. clusterName), and we always add it to the headers of each record. Makes sense to me :+1:
Yes, we were discussing this use case internally again yesterday.
A new config property that holds a sub-map of static key-value pairs would be ideal for now. We would use this to include a cluster name/address as a header with messages, and perhaps an indicator if the cluster is currently is the primary one where we are starting new workflow instances.
Hello everyone! If it is still free issue, I would like to take it and resolve. I think I could be the first step at contributing in an open source project. And one more question. @npepinpe said that it has to be one more issue in Zeebe project that can extend the exporter API. Has not that issue already opened? If it has not opened, I can open it and try to resolve also.
@npepinpe Hi Nicolas, Could you tell me, please, who is a maintainer for this project for now? Anyone could help me release my changes when they will be complete? Thank you for answer in advance.