KAFKA-17293: New consumer HeartbeatRequestManager should rediscover disconnected coordinator

[frankvicky]

JIRA: KAFKA-17293

Ensure that the new consumer will rediscover the coordinator on disconnect and send heartbeats to the correct coordinator.

Committer Checklist (excluded from commit message)

• [ ] Verify design and implementation
• [ ] Verify test coverage and CI build status
• [ ] Verify documentation (including upgrade notes)

[lianetm]

Hey @frankvicky , thanks for the nice patch! Looks good overall, just left some comments for consideration.

[frankvicky]

Hi @lianetm I have addressed the comments, PTAL 😺

[dajac]

@lianetm @frankvicky I wonder if we should also do the same on timeouts. Did we in the old implementation?

[lianetm]

I wonder if we should also do the same on timeouts. Did we in the old implementation?

Good point @dajac . It's not explicitly handled in the classic or new consumer, but we do get the behaviour in both, because timeouts as treated as disconnections at the network layer level (handleTimedoutRequests). So if the request times out, the network client disconnects from the node, ending up in the DisconnectExeption that both consumer handle to mark the coordinator unknown.

[lianetm]

Hey @frankvicky , thanks for the updates. Left couple of minor comments, and could you please check the test failures on the CommitRequestManagerTest? Makes sense that it probably needs tuning after the refactoring for handleCoordinatorDisconnect . Thanks!

[lianetm]

Hey @frankvicky , thanks for the updates. I left a couple of other minor comments. Thanks!