[Bug]: Unable to connect to GMK topic using Kafka-client 4.0.0

[JayajP]

What happened?

I tried running a Dataflow job that reads messages from a Kafka topic using kafka-client 4.0.0 and my reads were silently failing. Downgrading to 3.7.0 caused reads to succeed.

When I created my Kafka Read DoFn I specify withGcpApplicationDefaultCredentials

I'm currently running beam version 2.70.0-snapshot

c.c. @johnjcasey

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• [ ] Component: Python SDK
• [x] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [ ] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Infrastructure
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [x] Component: Google Cloud Dataflow Runner

[an2x]

@JayajP do you have any logs / error messages / stack trace to share?