Kafka External Stream to support write the headers
Use case
In #899 , we introduced the virtual column _tp_message_headers to read the headers as a map of string/string.
We also need to provide ways to write custom headers to Kafka. To comply with SQL INSERT syntax, the _tp_message_headers column probably need to be defined in the create external stream DDL.
This may not be implemented at high priority. If anyone has a strong use case, please leave comment below.
Describe the solution you'd like
Describe alternatives you've considered
Additional context
This will be implemented exactly the same way as how _tp_message_key is implemented (in enterprise version) for writing: https://docs.timeplus.com/proton-kafka#messagekey .
I.e. firstly, create an external stream with specifying the column ( must use map(string, string) ):
CREATE EXTERNAL STREAM example (
a int,
b string,
_tp_message_headers map(string, string)
) SETTINGS type='kafka', ...;
Then construct the headers when inserting data:
INSERT INTO example (a, b, _tp_message_headers) VALUES (1, 'one', {'key_1': 'value_1', 'key_2': 'value_2'});
This is backported in 3.0