snowflake-kafka-connector icon indicating copy to clipboard operation
snowflake-kafka-connector copied to clipboard

Key Parsing Rough Edges

Open GRVYDEV opened this issue 2 years ago • 1 comments

Hello, I have recently configured snowflake-kafka-connector a couple of times and i noticed that the key parsing is quite fragile. I am curious if there is a way to potentially make the parsing clean up newlines in the key. Essentially before actually parsing the key, iterate through it and check if there is a valid ascii line after each newline character. If so, then remove the newline character. This may make it easier to get setup since I had quite a bit of trouble figuring out the root cause of the parse errors.

GRVYDEV avatar Jun 16 '22 15:06 GRVYDEV

@GRVYDEV out of interest, are you referring to 'rough edges' like in the RECORD_METADATA where the key seems to have escaped unicode characters before it? For example (edited for sake of brevity):

{
  ...
  "key": "\u0000\u0000\u0000\u0000\u0018\u00148f14e45fce",
  "offset": 6,
  "partition": 0,
  ...
}

Feels like it's an escape sequence of some sort but not seeing any references to this issue, specifically.

JonathonO avatar Nov 15 '22 15:11 JonathonO