CTRL-d

[Wojdom]

Hello, I wonder if can you help me out with understanding kafkacat in producer mode.. Basically, the problem is why do i have to press ctrl-d to see messages in consumer? Any ideas? I just open one terminal with command: kafkacat -b localhost:9092 -t test -P

And the other one: kafkacat -b localhost:9092 -t test -C

Is it related to some kind of transactions? Previously I did not have such problems, all I was doing was typing message and pressing enter (new line).

[pszymczyk]

Hi

I have the same issue after migration to 1.7.0. On Version 1.6.0 (JSON, Avro, Transactions, librdkafka 1.7.0 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer) it works like a charm and on Version 1.7.0 (JSON, Avro, Transactions, IncrementalAssign, librdkafka 1.8.2 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer) I need to press ctrl-d to send messages.

[DuncanDoyle]

Same problem here as well. Try to send first message ... nothing happens. Press ctrl-D, message is sent. Try to send second message .... message is sent (without having to press ctrl-D again). I first wondered if it was some kind of transaction demarcation, but since after pressing ctrl-D once, sending of following messages just works, that doesn't seem to be the case.

This behaviour still exists on master (built today). Would like to know if this is intentional ...

[edenhill]

This is most likely because of stdio buffering. I.e., stdio won't release the line to kcat until a newline is seen.

You could try adding a setbuf(stdin, NULL); to kcat.c and see what happens

[k-wall]

I'm seeing this behaviour too:

Version 1.7.1 (JSON, Avro, Transactions, IncrementalAssign, JSONVerbatim, librdkafka 1.8.2 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer)

I'm running:

kcat -P -t my-topic -b kroxylicious-service:9292

then sending lines of input interactively to stdin:

foo
bar
baz

It is not until I strike ^D does kcat actually actually send three records. It is as if kcat producer is batching the records.

I tried passing options like -X linger.ms=1 -X batch.size=1 to the kcat producer, but that has no effect.

By adding the -d all I could see that kcat does actually seem to react at all until ^D is sent.

foo
bar
naz
<--- ^D struck
%7|1689937653.812|PARTITIONER|rdkafka#producer-1| [thrd:app]: my-topic [0] is the new sticky partition
%7|1689937653.812|TOPPAR|rdkafka#producer-1| [thrd:kroxylicious-service:9293/0]: kroxylicious-service:9293/0: my-topic [0] 3 message(s) in xmit queue (3 added from partition queue)

I tried running stdbuf -i0 -o0 -e0 kcat.. but that made no difference.

kafkacat 1.5.0 didn't have this issue.

foo
%7|1689938322.736|TOPPAR|rdkafka#producer-1| [thrd:kroxylicious-service:9293/0]: kroxylicious-service:9293/0: my-topic [0] 1 message(s) in xmit queue (1 added from partition queue)