Kafka (internal) logs via python API - like Java

[puneet336]

Hi, When i run java consumer code, i get lot of informative/kafka internal logs out of the box in addition to application logs ( via logging package or println ) on stdout, example -

[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Request joining group due to: group is already rebalancing
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] (Re-)joining group
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully joined group with generation Generation{generationId=13, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully synced group in generation Generation{generationId=13, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Updating assignment with
	Assigned partitions:                       [first_topic-0]
	Current owned partitions:                  [first_topic-0, first_topic-1]
	Added partitions (assigned - owned):       []
	Revoked partitions (owned - assigned):     [first_topic-1]

[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Revoke previously assigned partitions first_topic-1
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Request joining group due to: need to revoke partitions [first_topic-1] as indicated by the current assignment and re-join
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Notifying assignor about the new Assignment(partitions=[first_topic-0])
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Adding newly assigned partitions: 
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] (Re-)joining group
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully joined group with generation Generation{generationId=14, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully synced group in generation Generation{generationId=14, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Updating assignment with
	Assigned partitions:                       [first_topic-0]
	Current owned partitions:                  [first_topic-0]
	Added partitions (assigned - owned):       []
	Revoked partitions (owned - assigned):     []

[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Notifying assignor about the new Assignment(partitions=[first_topic-0])
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Adding newly assigned partitions: 

but with python consumer api, i dont get these messages on stdout. I get content from stdout via logging module and print functions. Is it possible to get these (kafka) internal logs when using python API?

confluent_kafka version=1.8.2 python version = 3.8.10. OS: linux mint 20.1

[jliunyu]

Hi @puneet336, thanks for asking.

The above logs you mentioned from java consumer are java clients logs, for example: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L609

For python clients, there is similar logs either. If you turn on the debug logs(Add "debug": "all" to your config), you will see the similar logs.

[edenhill]

And logs/debugs are printed to stderr, not stdout, as not to interfere with the program output.