proton icon indicating copy to clipboard operation
proton copied to clipboard

Kafka External Stream to support write the headers

Open jovezhong opened this issue 1 year ago • 1 comments

Use case

In #899 , we introduced the virtual column _tp_message_headers to read the headers as a map of string/string.

We also need to provide ways to write custom headers to Kafka. To comply with SQL INSERT syntax, the _tp_message_headers column probably need to be defined in the create external stream DDL.

This may not be implemented at high priority. If anyone has a strong use case, please leave comment below.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

jovezhong avatar Feb 19 '25 16:02 jovezhong

This will be implemented exactly the same way as how _tp_message_key is implemented (in enterprise version) for writing: https://docs.timeplus.com/proton-kafka#messagekey .

I.e. firstly, create an external stream with specifying the column ( must use map(string, string) ):

CREATE EXTERNAL STREAM example (
  a int,
  b string,
  _tp_message_headers map(string, string)
) SETTINGS type='kafka', ...;

Then construct the headers when inserting data:

INSERT INTO example (a, b, _tp_message_headers) VALUES (1, 'one', {'key_1': 'value_1', 'key_2': 'value_2'});

zliang-min avatar Feb 21 '25 04:02 zliang-min

This is backported in 3.0

yuzifeng1984 avatar Oct 23 '25 01:10 yuzifeng1984