chatmail
Too many messages sent per SMTP connection, leading to delayed delivery
- Operating System (Linux/Mac/Windows/iOS/Android): All
- Delta Chat Version: 1.30 (but probably all)
- Expected behavior: Recycle SMTP connections to avoid triggering per-connection message sending limits
- Actual behavior: SMTP connections are long-lived and many messages are sent down a single connection
- Steps to reproduce the problem:
- Start with a default-configured exim4 MTA + deltachat
- Send 20 or so messages in quick succession
- Note some of them aren't delivered immediately - depending on the queue runner, could be delayed by 15 minutes. You can observe this on the exim server (mailq) or just by watching them pop up at the receiver's side late and out-of-order.
This is something I come across, fix, and forget about, every time I re-configure my (exim4) MTA, so I figured it was time to get it into an issue: Exim's smtp_accept_queue_per_connection plays badly with deltachat. The default value for that setting is 10. https://www.exim.org/exim-html-current/doc/html/spec_html/ch-main_configuration.html#SECID105
The first 10 emails sent in an SMTP connection get a delivery attempt immediately, but after those first ten, messages are spooled to the queue to be delivered in the regular exim4 queue run - which might be 15 minutes later, depending on how the queue runner is configured.
For a deltachat user, this is quite jarring - depending on how fast you're sending messages, and how fast delta churns through SMTP connections, every now and then a message or two will seem to be unaccountably delayed by many minutes. This interrupts conversations and makes it feel like delta is unreliable.
Setting smtp_accept_queue_per_connection = 0 (if you know what you're doing and are sure it's fine in your case) makes Exim act in a more delta-friendly way, but given that this is a default setting and delta is unusually reliant on synchronous, rather than queued, mail delivery, perhaps it's worth putting in a heuristic or configuration setting into core, to recycle SMTP connections after so many messages have been sent down them, to avoid triggering the behaviour when a default delta meets a default exim?
If nothing else, having this issue in the history will help me next time I need to set up a new MTA :sweat_smile:
Other MTAs or SMTP-as-a-service providers might have similar per-connection limits,
I guess the intervention would be here: https://github.com/deltachat/deltachat-core-rust/blob/master/src/smtp.rs#L187=
There's inactivity-based staleness for SMTP connections but no number-of-messages-sent logic at the moment.
https://docs.rs/async-smtp/0.1.0/async_smtp/smtp/enum.ConnectionReuseParameters.html
smtp::ConnectionReuseParameters::ReuseLimited(10) would be lovely :sweat_smile:
Here is a related Server Fault question: https://serverfault.com/questions/893965/exim-configuration-on-ubuntu-16-04-queue-per-connection
ConnectionReuseParameters is now gone from async-smtp library and connections are managed by the deltachat core itself, so if we implement reconnection every 10 messages, we have to do it ourselves. I am not sure about doing it by default, because it definitely makes performance worse for bots using the servers without such limits, and to some extent normal users. Too frequent reconnections may trigger other limits too.
I will put this to project resurrection for now, until we find out that there is some well-known public server running with such configuration.
