confluent-kafka-python
confluent-kafka-python copied to clipboard
Kafka (internal) logs via python API - like Java
Hi, When i run java consumer code, i get lot of informative/kafka internal logs out of the box in addition to application logs ( via logging package or println ) on stdout, example -
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Request joining group due to: group is already rebalancing
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] (Re-)joining group
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully joined group with generation Generation{generationId=13, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully synced group in generation Generation{generationId=13, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Updating assignment with
Assigned partitions: [first_topic-0]
Current owned partitions: [first_topic-0, first_topic-1]
Added partitions (assigned - owned): []
Revoked partitions (owned - assigned): [first_topic-1]
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Revoke previously assigned partitions first_topic-1
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Request joining group due to: need to revoke partitions [first_topic-1] as indicated by the current assignment and re-join
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Notifying assignor about the new Assignment(partitions=[first_topic-0])
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Adding newly assigned partitions:
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] (Re-)joining group
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully joined group with generation Generation{generationId=14, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Successfully synced group in generation Generation{generationId=14, memberId='consumer-JJ2-1-19af6da5-85d1-4873-879b-c75c35ffbe2e', protocol='cooperative-sticky'}
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Updating assignment with
Assigned partitions: [first_topic-0]
Current owned partitions: [first_topic-0]
Added partitions (assigned - owned): []
Revoked partitions (owned - assigned): []
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Notifying assignor about the new Assignment(partitions=[first_topic-0])
[main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-JJ2-1, groupId=JJ2] Adding newly assigned partitions:
but with python consumer api, i dont get these messages on stdout. I get content from stdout via logging module and print functions. Is it possible to get these (kafka) internal logs when using python API?
confluent_kafka version=1.8.2 python version = 3.8.10. OS: linux mint 20.1
Hi @puneet336, thanks for asking.
The above logs you mentioned from java consumer are java clients logs, for example: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L609
For python clients, there is similar logs either. If you turn on the debug logs(Add "debug": "all" to your config), you will see the similar logs.
And logs/debugs are printed to stderr, not stdout, as not to interfere with the program output.