Feature: Configurable output format in connector SDK

[lovromazgon]

Feature description

The destination connector is responsible for deciding how a record gets encoded and sent to the 3rd party system. This means a user generally can't control what the connector produces (unless that connector implements different options like the S3 connector, see option format).

We should add functionality to the SDK that controls what gets produced by the connector. The idea is to tap into the sdk.Record struct and provide a function that marshals it into a byte slice (e.g. Record.Bytes()). The SDK could then inject an option in the connector config that controls the behavior of this method. For example the user could choose that Bytes() returns the JSON representation of the whole record, a Kafka Connect compatible JSON record, or just the payload. Connectors that produce raw data would then be expected to use the Bytes() method to marshal the record.

Note that connectors that mutate entities in place only work with structured data and would still be able to manually inspect structured data.

[neovintage]

This is about being a nicety in the SDK. Connector developers would be able to call a function and then all of the format changes could be made on their behalf. They wouldn't have to program all of that themselves. This is a fairly big issue.

[uchennakevinm1]

push to the next release