Processing Avro messages with embedded schema

[ttrading]

Hello:

I converted my raw array of JSON messages to an Avro message with embedded schema. How can I configure the sink to process them?

Thanks,

[willyborankin]

If you case is store data in PgSQL with JSON column type, then it is should be done like this:

1. Create a table with JSON column type in PgSQL database

CREATE TABLE your_topic_name (
  id INT PRIMARY KEY NOT NULL,
  json_value JSON NOT NULL,
  jsonb_value JSONB NOT NULL,
  uuid_value UUID NOT NULL
 )

2. Your Avro schema should look like this:

{
   "type": "record",
   "name": "pg_sql_types",
   "fields": [
          {"name": "id", "type": "int"},
          {"name": "json_value",  "type": "string"},
          {"name": "jsonb_value",  "type": "string"},
          {"name": "uuid_value",  "type": "string"} 
  ]
}

So in other words, connector can save JSON and UUID in PgSQL only if they define as a string type in your Avro schema and in a destination table PgSQL column type is UUID or JSON/JSONB. The reason is that Connector Schema does not support JSON type, only simple types and complex types like: Map, Struct, Array.

[ttrading]

Thank you for the explanation. I have a couple of more question please. I can either send my messages to Kafka in JSON with schema or AVRO with schema. Can you please let me know what converter settings should I be using in the standalone kafka-connect file for these two cases. The options I am talking about are "key.converter" and "value.converter". If I am including the schema in each AVRO message do I need a schema registry with the JDBC sink connector? Thanks

[willyborankin]

Sorry for the late answer. You can use same converter for "key" and "value". If you want to support schema versioning you need to use Confluent Schema Registry or Karapace