Asyncronous delivery confirmation

[antoinerabany]

Hi,

First, thanks for the work.

We are experiencing some slow-down while enqueuing message with confirm-delivery. Some time before we were not using this feature and enqueuing would be quite fast, and after enabling this feature, enqueuing would become quite slower.

I checked and it seems that the lib wait for the confirmation on each message while on the following blog post (https://blog.rabbitmq.com/posts/2011/02/introducing-publisher-confirms) they seems to tell us to enqueue all the message and then wait for all the confirmations, do you think there would be a way for amqpstorm to handle that ? If you have some indications, I could try an implementation.

Thank you.

[eandersson]

There are some fundamental problems with how I implemented RPC that will make this a bit tricky to implement, but shouldn't be impossible. I'll need some time to think about it.

However, if you are simply looking at publishing multiple messages you could just wrap the messages you are trying to publish around a transaction, e.g.

with channel.tx:
    for index in range(100):
        channel.basic.publish(....)

This would allow you to publish 100 messages fast and then rollback in case something goes wrong.

[jhogg]

Antoine,

Setting aside Erik's response on the complexities, I am curious as to your exact use case for this over transactions that he presented.

The example provided in the link is woefully incomplete and the Rabbit feature only adds a marginally incremental guarantee to delivery, which may not outweigh the application complexity to deal with the failures.

In full transaction mode, it is all-or-nothing, end to end. Recovery is relatively simple.

In the publisher-ack model with async out-of-order responses, here are a few things not in that example to consider:

• The example only tracks a failed publish by the publisher id, when in reality you need to know the exact data that failed to publish;
• There is no guarantee (actually, unlikely) that a re-publish will fix it. What is the recovery path with other messages in flight?
• There is no concept in the example of a message having a publish timeout, which is needed to not hang. Unexpected issues happen when live.
• A send() could raise an exception from a NAK of a prior message, meaning the current message didn't send but you are in a recovery path.
• Ack/nak comes back on a different thread, which is probably Erik's reference to how RPC is implemented. There is some juggling, and ultimately a "wait" against the connection necessary to ensure everything publishes/fails AND the exception is raised correctly on the right connection/thread.

The publisher-confirm is interesting in concept, but has some challenges, and I just wanted to make sure you were aware of them before heading too far down the path.

Jay

On Tue, Mar 15, 2022 at 4:02 AM Antoine R. @.***> wrote:

Hi,

First, thanks for the work.

We are experiencing some slow-down while enqueuing message with confirm-delivery. Some time before we were not using this feature and enqueuing would be quite fast, and after enabling this feature, enqueuing would become quite slower.

I checked and it seems that the lib wait for the confirmation on each message while on the following blog post ( https://blog.rabbitmq.com/posts/2011/02/introducing-publisher-confirms) they seems to tell us to enqueue all the message and then wait for all the confirmations, do you think there would be a way for amqpstorm to handle that ? If you have some indications, I could try an implementation.

Thank you.

— Reply to this email directly, view it on GitHub https://github.com/eandersson/amqpstorm/issues/116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJGJV3EQRMQACPML6BMZTVABGZBANCNFSM5QX7WUMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[antoinerabany]

Thanks for both your responses,

First I understood from the same article that transactions were quite slow. But It would fit my use case, so I will try to implement them and see if there is performance degradation.

Thank you again !

[eandersson]

Thanks for both your responses,

First I understood from the same article that transactions were quite slow. But It would fit my use case, so I will try to implement them and see if there is performance degradation.

Thank you again !

Sounds good! Let me know if you run into any performance issues!