[FLINK-28060][BP 1.15][Connector/Kafka] Updated Kafka Clients to 3.1.1

[MartijnVisser]

This is a backport of https://github.com/apache/flink/pull/19994

Will merge this PR after the Flink 1.15.1 release has been completed, as proposed in https://lists.apache.org/thread/pns8z69mqbqwgbs7l50n9c9k522f649g

[flinkbot]

CI report:

• 401bb4d5e5246ad6a0d765cb95125a78ffb25a61 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[LinMingQiang]

I still have problems applying this patch on 1.15.

[MartijnVisser]

What is the problem you're having?

[LinMingQiang]

After modification, only another error will be thrown.

Caused by: org.apache.flink.runtime.taskmanager.AsynchronousException: Caught exception while processing timer. at org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1535) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1510) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1650) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$21(StreamTask.java:1639) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262] Caused by: org.apache.flink.streaming.runtime.tasks.TimerException: java.lang.NullPointerException: timestamp must to be set before retrieving it. at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1650) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$21(StreamTask.java:1639) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262] Caused by: java.lang.NullPointerException: timestamp must to be set before retrieving it. at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:76) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.connectors.kafka.table.ReducingUpsertWriter$WrappedContext.timestamp(ReducingUpsertWriter.java:171) ~[flink-sql-connector-kafka-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.connector.kafka.sink.KafkaWriter.write(KafkaWriter.java:199) ~[flink-sql-connector-kafka-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.connectors.kafka.table.ReducingUpsertWriter.flush(ReducingUpsertWriter.java:146) ~[flink-sql-connector-kafka-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.connectors.kafka.table.ReducingUpsertWriter.lambda$registerFlush$3(ReducingUpsertWriter.java:124) ~[flink-sql-connector-kafka-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1648) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$21(StreamTask.java:1639) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist-1.15.1-vip.jar:1.15.1-vip] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262]

[MartijnVisser]

Are you sure that is related to this patch? This patch is only updating the Kafka Clients and the CI is passing. The stacktrace makes it appear that there's something wrong with your Flink application.

[LinMingQiang]

I'm not sure. The problem occurred after upgrading 3.1.1.

[MartijnVisser]

We're not merging this at this point. If you also have this problem on the master branch (since we've upgraded there), please create a Jira ticket for it.