Producer.publish blocks forever on a closed connection when the transport_options of the connection does not specify a retry_policy

[danidee10]

Kombu version = 4.4.0

When I call publish on a closed connection, it blocks forever; No exception, No error messages whatsoever. I tried to set a retry policy but The behaviour is the same.

Reproduce

from kombu import Connection

with Connection('pyamqp://localhost:8000') as conn:
    producer = conn.Producer(serializer='json')
    producer.publish(
        'Hello', errback=lambda: print('An error occured!'),
        retry=True, retry_policy={
            'interval_start': 0,
            'interval_step': 1,
            'interval_max': 5,
            'max_retries': 3,
        }
    )

    print('Published!')  # Never gets called

There's no Rabbitmq server on localhost:8000 So I expect publish to raise a socket.timeout error when it's unable to connect. Instead, it just hangs forever because it's trying to establish a connection.

[thedrow]

This is a somewhat annoying design flaw.

The code in question eventually invokes: https://github.com/celery/kombu/blob/77b1362b39665b79afba64b501a94fa7e730af16/kombu/connection.py#L827-L855

That ensure_connection() call uses the transport_options of the connection instead of the retry_policy you provided.

The following code behaves as expected:

from kombu import Connection

with Connection('pyamqp://localhost:8000',
                transport_options={
                    'retry_policy': {
                        'interval_start': 0,
                        'interval_step': 1,
                        'interval_max': 5,
                        'max_retries': 3,
                    }
                }) as conn:
    producer = conn.Producer(serializer='json')
    producer.publish(
        'Hello', errback=lambda: print('An error occured!'),
        retry=True, retry_policy={
            'interval_start': 0,
            'interval_step': 1,
            'interval_max': 5,
            'max_retries': 3,
        }
    )

    print('Published!')  # Never gets called

I'm guessing this problem won't be resolved any time soon as it has a workaround. In the meanwhile, it should be documented that you probably want to specify both. Care to issue a PR?

[thedrow]

This is a regression from https://github.com/celery/kombu/commit/816e3dcf346decd4d458e2a088ac7a914b48dcce.

[danidee10]

Thanks for the response. I tried it and it works :+1: Yes I'll open a PR soon!

[srikiraju]

@danidee10 Any progress on this issue?

[danidee10]

Hi, @srikiraju I've been really busy. I will try to open a PR (Documentation) this weekend

[danidee10]

I don't think the Documentation is necessary (I was unaware of this PR: https://github.com/celery/kombu/pull/1026 when I posted the last message)

[matteius]

@danidee10 In the original report you are testing this behavior outside of Celery correct? That is really cool and perhaps you wouldn't mind submitting a PR with a unit test that shows this failure -- we need more unit test coverage here regardless so if you can cover that regression with a failing test in your version of kombu 4.4.0 while in parallel test if the latest 4.6.3 and 4.6.4 (I am less likely to recommend 4.6.4, esp for folks using redis due to a different bug report, but try both) and also making a PR with such a unit test into celery/kombu will kick of the Travis Ci and Appveyor CI checks against master and so we can really pinpoint that it stays fixed and isn't reintroduced as a regression later on.

[danidee10]

@matteius I just opened a PR and I can confirm that the test fails in 4.4.0, 4.6.3 and 4.6.4 and it works in #1026

PR link: https://github.com/celery/kombu/pull/1093

[matteius]

Awesome! -- I am wrapped up this weekend in some outdoor projects but I'll help take a look in a few days. Have a great Labor Day weekend.