Producer hide librdkafka failures and silently return None as message on poll

[Hubbitus]

trafficstars

Description

In case an error happened on Kerberos request authentification librdkafka will provide some logs which also are not returned into python space and for just None returned from message poll without an error indication.

How to reproduce

consumer_config = {
                    'bootstrap.servers': 'kafka-int.epm-eco.projects.example.com:9095}',
                    'schema.registry.url': 'http://schema-registry-int.epm-eco.projects.example.com:8081',
                    'group.id': 'epm-ddo.consumer.2',
                    'auto.offset.reset': 'earliest',
                    'security.protocol': 'SASL_SSL',
                    'ssl.ca.location': 'conf/integration/epm-eco-int.ca.crt',
                    'sasl.mechanisms': 'GSSAPI',
                    'sasl.kerberos.principal': 'kafkaclient',
                    'sasl.kerberos.kinit.cmd': 'bash -c "cat conf/integration/paswd | /usr/bin/kinit [email protected]"',
                    'sasl.kerberos.service.name': 'kafka',
#                    'log_level': 7, 'debug': 'all' ## Logs will be ONLY in container output
                  }

topic = '....cdm.LocationChartNode'

consumer = AvroConsumer(consumer_config)
consumer.subscribe([topic])

message = consumer.poll(5)
print(message)

consumer.close()

Printed result None, if look into the container, librdkafka produce errors like:

%3|1628784442.369|SASLREFRESH|rdkafka#consumer-34| [thrd:main]: Kerberos ticket refresh failed: bash -c "cat conf/integration/paswd | /usr/bin/kinit [email protected]": exited with code 1

Checklist

Please provide the following information:

• [✔] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): ('1.7.0', 17235968), ('1.7.0', 17236223)
• [✔] Apache Kafka broker version: 5.3.2 (uncertain)
• [✔] Client configuration: Kerberros (SASL_SSL), see full config above in reproduce section
• [✔] Operating system: Linux. Fedora and Ubuntu checked
• [✔] Provide client logs (with 'debug': '..' as necessary)
• [✔] Provide broker log excerpts: sems irrelevant andI have no access
• [✔] Critical issue: yes. Silently skip errors behavior do not allow rely on behaviour and understand what is there something go wrong or just end of topic reached

[jliunyu]

@Hubbitus, thanks for reporting this. Usually the error_cb can capture the error, please try to use it. This is the example: https://github.com/confluentinc/confluent-kafka-python/blob/master/tests/test_misc.py#L44, if the failure still can't be captured, please feel free to let us know and we will take a look.

[Hubbitus]

Some errors indeed may be captured, I've tried that indirectly. Meantime, there should not be success return from poll on fatal errors, is not?