kafka icon indicating copy to clipboard operation
kafka copied to clipboard

KAFKA-13818: Add generation to consumer assignor logs

Open vvcephei opened this issue 3 years ago • 3 comments

Reading assignor logs is really confusing in large part because they are spread across different layers of abstraction (the ConsumerCoordinator and the ConsumerPartitionAssignor, which in Streams consists of several layers of its own). Each layer in the abstraction reports useful information that only it has access to, but because they are split over multiple lines, with multiple members in the cluster, and (often) multiple rebalances taking place in rapid succession, it's often hard to understand which logs are part of which rebalance.

This PR attempts to improve the situation by adding a common prefix to all assignment logs that includes the current generation number. Then, debuggers can pull out all logs for a particular rebalance with a simple grep "generationId=5". In order to add this useful context without violating encapsulation, the PR simply adds a generic "dynamic log context" that assignors can optionally participate in (by mixing in ContextualLogging).

I chose to keep these changes restricted to private APIs for the moment. If this approach proves useful, we can propose a KIP at a later date to make the APIs public.

Committer Checklist (excluded from commit message)

  • [ ] Verify design and implementation
  • [ ] Verify test coverage and CI build status
  • [ ] Verify documentation (including upgrade notes)

vvcephei avatar Apr 08 '22 18:04 vvcephei

Build failed. Looks concerning?

mjsax avatar Apr 08 '22 23:04 mjsax

Thanks John, agree that generation ID would be useful to add to help group messages for a single rebalance from different loggers together. The duplication and boilerplate in here don't seem great, though--I've opened up https://github.com/apache/kafka/pull/12023 as a PR against this branch to take a stab at addressing that. What do you think?

C0urante avatar Apr 09 '22 17:04 C0urante

Note: I might want to adjust the approach here, as in a subsequent investigation, I have realized it would be quite nice to prefix the cluster membership lines with the generation id as well, but we would not want to include the entire log prefix on each line (I don't think we would anyway).

See https://issues.apache.org/jira/browse/KAFKA-14253 for the cluster member lines I'm thinking of.

vvcephei avatar Sep 21 '22 22:09 vvcephei