Two groups accidentally created while fetching pending messages after backup restore

[gerryfrancis]

• Operating System (Linux/Mac/Windows/iOS/Android): Android 11.

• Delta Chat Version: 1.44.0 (nightly built 2024-03-26).

• Expected behavior: When pending messages are fetched from the Inbox and Sent IMAP folders after a backup has been restored, emails that were sent by MUAs, e.g. Webmail clients, and contain Re: in the first place of the subject, e.g. in replies, do not trigger creating a new group named Re: [Subject].

• Actual behavior: When pending messages are fetched from the Inbox and Sent IMAP folders after a backup has been restored, emails that were sent by MUAs, e.g. Webmail clients, and contain Re: in the first place of the subject, e.g. in replies, trigger creating a new group named Re: [Subject].

• Steps to reproduce the problem: -- Create a backup of your Delta Chat account. -- Un- and reinstall Delta Chat, but do not launch the app, yet. -- Receive a classic email that has been addressed to you and a second recipient. (The email should remain in the Inbox folder of your IMAP email account.) -- Reply to that email by using a MUA client, e.g. Webmail. (Your email should be stored in the Sent folder with Re: [Subject] as the subject.) -- Launch Delta Chat and restore the backup of your account. (Result: Two groups are created, named Re: [Subject] (which contains the reply message) and [Subject] (which contains the first message), while pending messages are being fetched.)

• Screenshots:

• Logs: N/A.

[gerryfrancis]

FYI, it happens with version 1.45.0 (beta), too.

[iequidoo]

A fix i can imagine here is to sort by INTERNALDATE not only messages from a particular folder, but from all folders, before passing them to receive_imf_inner(). The problem is that folders are processed by different threads, so it will be actually not sorting, but some kind of synchronisation using e.g. tokio::sync::Notify + Mutex. Not too difficult to implement, but still quite a big change so it's a question whether fixing this is worth it. Maybe also a topo-sorting using the References header is a way.

[iequidoo]

Another possible fix is to leave this unordered processing of folders as is, but search Message-ID in the already known references, this way we can insert an unordered ("missed previously") message to the already existing chat.

EDIT: the problem is that several messages may be unordered, in this case we know the proper chat for them only when processing the last unordered message. Merging chats looks too complicated to fix this, some INTERNALDATE sorting-based solution could be better.

EDIT: A better possible fix: even if a message doesn't reference any known message, but has the same subject (without prefix) and "group members" as an already existing ad-hoc group, use that group for the new message.

[gerryfrancis]

@iequidoo I did the test with the latest nightly build: Those messages are assigned to one group now - thank you very much! :+1: :) The only problem left is the chronological order, as the reply/replies (stored in the Sent folder) seem(s) to be fetched/assigned to the group first, but this issue might not be solved easily. Anyway, thanks a lot for the great work! :)

Edit: Oh wow, I have found the PR in regard to the chronological ordering: https://github.com/deltachat/deltachat-core-rust/pull/5800 Thank you so much for the heads up! :)

[iequidoo]

Edit: Oh wow, I have found the PR in regard to the chronological ordering: #5800

Unfortunately, this PR won't help in your scenario, it only sorts new received outgoing messages down which is useful for multi-device, particularly when an account is shared by users. There's another PR, #5813, that adds a test on your scenario, this test fails currently. But this can be fixed by introducing a new state for received outgoing messages so that they are sorted like fresh incoming ones, which i plan to do once #5800 is merged, mostly to avoid self-conflicts as that doesn't affect many users.