Schema Registry and Avro Format Deserialization Issues

[iankittomar]

I am encountering deserialization issues with the Kafka Delta Connector when using Avro format with a Schema Registry, and when using Avro/JSON without a Schema Registry.

With Schema Registry and Avro Format:

Error:

web_requests: Deserialization failed - partition 0, offset 0, dead_letter
Error: could not parse to RawRegisteredSchema, schema might not exist on this schema registry,
the HTTP call failed, cause will give more information,
was caused by error decoding response body: expected value at line 1 column 1,
it's retriable: false, it's cached: true

• I confirmed the schema exists in the registry, but it seems like there might be an issue with the HTTP call or the response parsing.

With Avro and JSON without Schema Registry:

Error:

Deserialization failed - partition 0, offset 0, dead_letter wrong magic in header 
Deserialization failed - partition 0, offset 2, dead_letter wrong magic in header 
Deserialization failed - partition 0, offset 3, dead_letter wrong magic in header

• This error occurs when attempting to deserialize Avro and JSON messages without using the Schema Registry.

Default (without specifying format, assuming JSON):

No issues encountered; data is deserialized and successfully written to Azure Data Lake Storage (ADLS) without any errors.