kombu
kombu copied to clipboard
celery
kombu silently catches OSError in consumer code
I'm running a ConsumerProducerMixin using kombu 4.1 with RabbitMQ 3.6 and noticed the following issue:
When logging is not configured (kombu.utils.debug.setup_logging is not called), kombu silently eats every exception that is a subclass of OSError (in my case urllib.error.HTTPError which is thrown by urllib). This is because Connection and thusConsumerMixin registers OSError in self.connection_errors which are then caught silently by ConsumerMixin.run. Although the silent catching is hard to debug, the real issue is that kombu catches Exceptions in user-ocde because it makes the flawed assumption that any OSError is a kombu connection error.
Sample code:
from urllib.error import HTTPError
from kombu.mixins import ConsumerProducerMixin
class Worker(ConsumerProducerMixin):
def __init__(self, connection, queue):
self.connection = connection
self.queue = queue
def get_consumers(self, Consumer, channel):
return [
Consumer(
queues=[self.queue],
on_message=self.on_request,
)
]
def on_request(self, message):
# 1: correct behavior
raise Exception("This exception is not caught")
# 2: wrong behavior, can be triggered by urllib functions
raise HTTPError("This exception is caught by kombu")
This is part of the connection retry mechanism. Each transport can declare a set of recoverable exception classes. Are you using librabbitmq ?
In this setup we never installed librabbitmq, so kombu should use py-amqp (which registers OSError aswell).
Our temporary solution is catching all exceptions in our consumer code to avoid propagating OSErrors to kombu. This also has the benefit that the connection is not dropped when an error occurs of which we know that the kombu connection is unaffected (in our example: HTTP call to another endpoint fails). Still, kombu catching all OSErrors from user code does seem strange to me.
Just hit this error. Using requests in my consumer and when I get an exception the messages just gets requeued and it all goes in a silent loop. Any updates on the plan to fix it?
