feat: Use IMAP APPEND command to upload sync messages (#5845)

[iequidoo]

NB: This doesn't yet implement setting of the \Seen flag when uploading a sync message, async_imap::Session::append() needs to be tweaked first.

[iequidoo]

W/o the seccond commit (0f6f3f1aa6d792849088de6fe98623b86258e041) tests/test_securejoin.py::test_qr_readreceipt fails: https://github.com/deltachat/deltachat-core-rust/actions/runs/10517506038/job/29141924140

DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/imap.rs:1119: Marked messages 6 in folder INBOX as seen.'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/imap.rs:525: No new emails in folder "INBOX".'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'ConnectivityChanged'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'ImapInboxIdle'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/scheduler.rs:691: IMAP session supports IDLE, using it.'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/smtp/send.rs:57: Message len=2047 was SMTP-sent to ci-n2msa7@***.'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'SmtpMessageSent', 'msg': 'Message len=2047 was SMTP-sent to ci-n2msa7@***'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/smtp.rs:583: Successfully sent MDN for Mr.nMKLo-gImM7.Xd1iWIWakvf@localhost.'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/smtp.rs:617: Sending MDNs.'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/scheduler.rs:804: SMTP fake idle started.'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'ConnectivityChanged'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/scheduler.rs:832: SMTP has no messages to retry, waiting for interrupt.'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'chatId': 10, 'kind': 'MsgDelivered', 'msgId': 22}
DEBUG    root:rpc.py:180 account_id=2 got an event {'chatId': 10, 'kind': 'ChatlistItemChanged'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'kind': 'Info', 'msg': 'src/imap.rs:525: No new emails in folder "INBOX".'}
DEBUG    root:rpc.py:180 account_id=3 got an event {'kind': 'Info', 'msg': 'src/imap/idle.rs:65: INBOX: Idle entering wait-on-remote state'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'kind': 'ConnectivityChanged'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'kind': 'ImapInboxIdle'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'kind': 'Info', 'msg': 'src/scheduler.rs:691: IMAP session supports IDLE, using it.'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'kind': 'Info', 'msg': 'src/imap.rs:1732: "INBOX": got unsolicited response Other(ResponseData { raw: 4096, response: Done { tag: RequestId("A0039"), status: Ok, code: Some(AppendUid(1724372145, [Uid(7)])), information: Some("Append completed (0.085 + 0.126 + 0.084 secs).") } })'}
DEBUG    root:rpc.py:180 account_id=2 got an event {'kind': 'Info', 'msg': 'src/imap/idle.rs:65: INBOX: Idle entering wait-on-remote state'}
---------------------------- Captured log teardown -----------------------------

It seems that a completed APPEND somehow affects receiving other messages (at least MDNs in this test, so Bob doesn't receive the MDN), i don't see unsolicited EXISTS in the log, so the Inbox loop goes sleep instead of refetching. But if i trigger refetching on APPEND responses, the test works. May be even a server-side issue because other unsolicited responses don't affect receiving MDNs.

[iequidoo]

From https://www.rfc-editor.org/rfc/rfc3501#section-6.3.11:

  If the mailbox is currently selected, the normal new message
  actions SHOULD occur.  Specifically, the server SHOULD notify the
  client immediately via an untagged EXISTS response.  If the server
  does not do so, the client MAY issue a NOOP command (or failing
  that, a CHECK command) after one or more APPEND commands.
  

So, the server SHOULD, not MUST. Maybe the chatmail server doesn't do so and that's why refetching Inbox after the APPEND response helps?

[link2xt]

Adding optional arguments for APPEND here: https://github.com/async-email/async-imap/pull/106

[iequidoo]

I made a dumb merge, but this actually breaks what was fixed in 418dfbf9946027567d598fa94e528c0ec44e34be because groups are promoted by SMTP, but QR code tokens are synced by IMAP. Need to think how to fix this.

EDIT: True, JSON-RPC tests are broken now: https://github.com/deltachat/deltachat-core-rust/actions/runs/10712806799/job/29704001842?pr=5901

[link2xt]

This race condition between getting a group promoted and receiving a sync message for its QR code is already a problem even without this PR if sync messages are moved to DeltaChat but group messages are not because these folders are fetched in arbitrary order. If second device fetches DeltaChat and gets sync message before fetching INBOX and receiving first group message, it is going to drop the QR code.

Maybe QR code token should be saved into tokens regardless of whether the group exists, but if we don't know about this group (yet) and cannot accept join requests, we should ignore join requests and let the first device handle them.

In the tests we need to artificially wait until QR code and group is synced before scanning the QR code.

[hpk42]

What is the reason to introduce "IMAP APPEND" at all? We typically try to use IMAP only in a minimal way.

On Tue, Aug 20, 2024 at 13:23 -0700, iequidoo wrote:

NB: This doesn't yet implement setting of the \Seen flag when uploading a sync message, async_imap::Session::append() needs to be tweaked first. You can view, comment on, or merge this pull request online at:

https://github.com/deltachat/deltachat-core-rust/pull/5901

-- Commit Summary --

• feat: Use IMAP APPEND command to upload sync messages (#5845)

-- File Changes --

M src/chat.rs (46)
M src/config.rs (4)
M src/imap.rs (47)
M src/mimefactory.rs (6)
M src/scheduler.rs (14)
M src/smtp.rs (1)
M src/sql/migrations.rs (14)
M src/sync.rs (22)
M src/test_utils.rs (44)

-- Patch Links --

https://github.com/deltachat/deltachat-core-rust/pull/5901.patch https://github.com/deltachat/deltachat-core-rust/pull/5901.diff

-- Reply to this email directly or view it on GitHub: https://github.com/deltachat/deltachat-core-rust/pull/5901 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

[iequidoo]

What is the reason to introduce "IMAP APPEND" at all? We typically try to use IMAP only in a minimal way.

I forgot to add the motivation to the commit message, done now:

• With IMAP APPEND we can upload messages directly to the DeltaChat folder (for non-chatmail accounts).
• We can set the \Seen flag immediately so that if the user has other MUA, it doesn't alert about a new message if it's just a sync message (there were several such reports on the support forum). Though this also isn't useful for chatmail.
• We don't need SMTP envelope and overall remove some overhead on processing sync messages.

Are there any server implementations not supporting APPEND? I think this may be the only blocker

[link2xt]

What is the reason to introduce "IMAP APPEND" at all? We typically try to use IMAP only in a minimal way.

Primary reason is avoiding sync messages appearing in the inbox before being moved into DeltaChat folder because user complain that they appear in the web interface: https://support.delta.chat/t/receiving-blank-emails-with-encrypted-attachments-whenever-an-email-blocked-deleted-or-accepted/3206 Gmail seems to keep the message displayed even some time after moving, so placing it directly into DeltaChat avoids this.

[link2xt]

Are there any server implementations not supporting APPEND?

Unlikely, most clients save sent messages into Sent folder using APPEND.

[link2xt]

I opened an issue regarding APPEND response parsing in async-imap, going to look into it: https://github.com/async-email/async-imap/issues/108

Edit: made a PR https://github.com/async-email/async-imap/pull/109

Edit2: published async-imap 0.10.1