Support messages in Avro format

[berndruecker]

Context:

• Kafka users often prefer Avro as the message format instead of JSON (was also recommended since the beginning: https://www.confluent.io/blog/avro-kafka-data/).
• Kafka Connect Zeebe is currently hardcoded to use JSON, which limits the applicability of the connector in some use cases
• Zeebe internally stores data as JSON (https://docs.camunda.io/docs/product-manuals/concepts/variables)

Now there are two possibilities:

1. Convert messages from/to Json/Avro when they come in/go out. Maybe transforms (https://github.com/camunda-community-hub/kafka-connect-zeebe/issues/18) can do this and there is no need to do changes in this connector
2. Extend the connector to be more flexible on this, for example in the sink it might be the https://github.com/camunda-community-hub/kafka-connect-zeebe/blob/master/src/main/java/io/zeebe/kafka/connect/sink/message/JsonRecordParser.java#L49 and in the source around https://github.com/camunda-community-hub/kafka-connect-zeebe/blob/177f1649c572b70fb709601a32ae552b4585c44c/src/main/java/io/zeebe/kafka/connect/source/ZeebeSourceTask.java#L143

We should first explore if "transforms" do the trick (in this case adding some documentation might be sufficient).

[berndruecker]

Googling around I found this interesting thread: https://groups.google.com/g/confluent-platform/c/OGuOCkEKoeA.

My summary:

• Easiest would be to use KSQL to convert Avro to JSON and vice versa before using the Converter
• It is possible to build a custom SMT (https://www.confluent.io/blog/kafka-connect-single-message-transformation-tutorial-with-examples/) to do this, but this does not exist out of the box yet (some inspiration can be found in https://github.com/ottomata/kafka-connect-jsonschema, https://github.com/epheatt/kafka-connect-jsonavroschema/blob/master/src/main/java/com/github/epheatt/kafka/connect/JsonToAvroSchemaConverter.java or probably also https://github.com/ewencp/kafka/commit/3abb54a8062fe727ddaabc4dd5a552dd0b465a03 or https://stackoverflow.com/questions/65517359/kafka-connect-custom-transforms-to-convert-schema-less-json-to-avro). This probably needs a bit more understanding of how SMT and formats work in Kafka.
• All of this would be independent of the connector itself