Externally deleted emails show up in the inbox and cannot be deleted

[LostinSpacetime]

Steps to reproduce

1. Open your email account in Thunderbird with new emails
2. Delete one or more emails before reading them
3. Open the same account in the Mail app on the cloud

Expected behavior

The inbox shouldn't contain the deleted emails.

Actual behavior

The inbox contains the deleted emails and the emails are marked as unread. The overall counter for unread emails is 0. If one clicks on one of the emails, instead of the content on the email, "not found" is shown. Trying to delete the emails results in failure with the message "Message could not be deleted" and after a few seconds the "ghost message" reappears in the inbox.

Mail app version

3.5.0

Mailserver or service

IMAP

Operating system

Ubuntu 18.04

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database

MariaDB

Additional info

No response

[LostinSpacetime]

Currently, my inbox is filled with non-existent emails, and is basically unusable. I would be thankful for a way to rescan my emails as a temporary workaround.

[3Nigma]

Same here

[alexh-swdev]

Same here:

• Mail 3.5
• NC 28.0.1
• PHP 8.2.14 (fpm w/ Apache 2.4.46)
• Debian bullseye

I tried the commands w/o success:

• mail:clean-up
• mail:sync

I could not figure out a rule which mails I can remove from my local MUA (which is emClient in my case) but remain in NC-Mail. But there should definetly be an easy way to remophe orphanded mails (mail which are not found) as a hotfix ASAP unless this is fixed properly

/Edit: I also have Redis and APCu active (but this is independent from the error, just tried it)

/Edit2: NC Log shows this:

[paulvt]

It is possible to use "Clear mailbox" on my Inbox, and then the messages are gone. However, because I also suffer from #5323, when I go to the Priority inbox, they are still all there! Then, when I mark them as Unimportant and reload the page/app, they are finally all gone.

So the issue seems to be purely that threads and prioritization tags keep tracks of messages that are gone.

[thrommel1980]

Same problem here. I already avoid deleting emails in the K9 Mail app. If I do, they are in my NC app as corpses. But are deleted in my mail provider inbox.

[mayermart]

I do not think this is a Problem caused by K9 itself. ~~After reading some issues and some experiments i got rid of the problem by simply disable the "Mark E-Mails as Important" Feature in the Nextcloud Mail Settings.~~

To delete all remaining (Ghost-)E-Mails from the Mailbox i

1. moved all actually existing Mails via Thunderbird (K9 would work the same i guess) in a new temporary Folder
2. used the "Clear mailbox" option on the Inbox folder in Nextcloud Mail and
3. moved all Mails from the temporary Folder back into my Inbox via Thunderbird.

Edit 2024-04-02: The Problem reappeared but less frequently.

[maximelehericy]

same is happening to me, with archived or sent to junk emails.

[violoncelloCH]

posting the full log lines in case that helps:

{"reqId":"YFRxUA0D1JXRPoiRvzhL","level":3,"time":"2024-01-22T20:56:55+00:00","remoteAddr":"172.22.0.4","user":"jonas","app":"mail","method":"DELETE","url":"/apps/mail/api/thread/36311","message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","version":"27.1.5.1","exception":{"Exception":"Exception","Message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334","Code":0,"Trace":[{"file":"/var/www/html/lib/private/AppFramework/App.php","line":183,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Mail\\Controller\\ThreadController"],"delete"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Mail\\Controller\\ThreadController","delete",["OC\\AppFramework\\DependencyInjection\\DIContainer"],["36311","mail.thread.delete"]]},{"file":"/var/www/html/lib/base.php","line":1068,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/mail/api/thread/36311"]},{"file":"/var/www/html/index.php","line":38,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","Line":169,"Previous":{"Exception":"TypeError","Message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned","Code":0,"Trace":[{"file":"/var/www/html/custom_apps/mail/lib/Service/MailManager.php","line":381,"function":"move","class":"OCA\\Mail\\IMAP\\MessageMapper","type":"->","args":[["OCA\\Mail\\IMAP\\ImapClientRateLimitingDecorator",["HICenv","HICflags","HIChdrs","HICdate","HICsize","HICstruct"],true,true],"INBOX",1582,"Trash"]},{"file":"/var/www/html/custom_apps/mail/lib/Service/MailManager.php","line":340,"function":"deleteMessageWithClient","class":"OCA\\Mail\\Service\\MailManager","type":"->","args":[["OCA\\Mail\\Account"],["OCA\\Mail\\Db\\Mailbox",8],1582,["OCA\\Mail\\IMAP\\ImapClientRateLimitingDecorator",["HICenv","HICflags","HIChdrs","HICdate","HICsize","HICstruct"],true,true]]},{"file":"/var/www/html/custom_apps/mail/lib/Service/MailManager.php","line":954,"function":"deleteMessage","class":"OCA\\Mail\\Service\\MailManager","type":"->","args":[["OCA\\Mail\\Account"],"INBOX",1582]},{"file":"/var/www/html/custom_apps/mail/lib/Controller/ThreadController.php","line":205,"function":"deleteThread","class":"OCA\\Mail\\Service\\MailManager","type":"->","args":[["OCA\\Mail\\Account"],["OCA\\Mail\\Db\\Mailbox",8],"<[email protected]>"]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"delete","class":"OCA\\Mail\\Controller\\ThreadController","type":"->","args":[36311]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Mail\\Controller\\ThreadController"],"delete"]},{"file":"/var/www/html/lib/private/AppFramework/App.php","line":183,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[["OCA\\Mail\\Controller\\ThreadController"],"delete"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Mail\\Controller\\ThreadController","delete",["OC\\AppFramework\\DependencyInjection\\DIContainer"],["36311","mail.thread.delete"]]},{"file":"/var/www/html/lib/base.php","line":1068,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/mail/api/thread/36311"]},{"file":"/var/www/html/index.php","line":38,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php","Line":334},"message":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334","exception":[],"CustomMessage":"OCA\\Mail\\IMAP\\MessageMapper::move(): Return value must be of type int, null returned in file '/var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php' line 334"},"id":"65aed6b3751ba"}
{"reqId":"YFRxUA0D1JXRPoiRvzhL","level":3,"time":"2024-01-22T20:56:55+00:00","remoteAddr":"172.22.0.4","user":"jonas","app":"PHP","method":"DELETE","url":"/apps/mail/api/thread/36311","message":"Undefined array key 1582 at /var/www/html/custom_apps/mail/lib/IMAP/MessageMapper.php#334","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0","version":"27.1.5.1","data":{"app":"PHP"},"id":"65aed6b3751f5"}

[meichthys]

Currently, my inbox is filled with non-existent emails, and is basically unusable. I would be thankful for a way to rescan my emails as a temporary workaround.

This is not a fix, but a workaround that usually helps (granted we should not need to run in debug mode to workaround this issue):

• Enable Debug mode in Nextcloud config. php:
  • 'debug' => true,
• In the Nextcloud Mail app, click the ... menu on the folder exhibiting the issue
• Choose Clear cache (Only available in debug mode)

[LostinSpacetime]

This is not a fix, but a workaround that usually helps (granted we should not need to run in debug mode to workaround this issue):

* Enable Debug mode in Nextcloud `config. php`:
  
  * `'debug' => true,`

* In the Nextcloud Mail app, click the `...` menu on the folder exhibiting the issue

* Choose `Clear cache` (Only available in debug mode)

Thank you! I will try this on the weekend, since I would rather not put the cloud in Debug mode on a working day. We are using Nextcloud in our school and several teachers are complaining about this issue. Is there a possibility to clear the cache for all or certain users?

[caltenhofen]

I agree, it would be great to have this as an option for all users. Or is there any risk in doing so?

[frederikb96]

I would also highly appreciate if the clear cache mode would be available without debug mode, thanks :)

[HLFH]

This issue is a duplicate of https://github.com/nextcloud/mail/issues/9048.

[alexh-swdev]

After skimming the other issue: No!

[caltenhofen]

I agree from my understanding it's probably a similar but distinct issue. But as I am not too familiar with the codebase the solutions might be connected.

[violoncelloCH]

I observed that this particularly happens for me for mails that get classified as "important" by Nextcloud Mail. Not sure if this is a necessary requirement, but for the last few occurrences I could observe it was always mails tagged as important that then stick in the Nextcloud Mail UI. Interesting is also, that I can still mark the mail as "not important" while all other actions don't work on the ghost mail. So maybe that's somehow related to how that "important" mail feature interacts with the mailbox?

[LostinSpacetime]

I want to add that even after turning off the priority emails in the app-settings, in Thunderbird deleted emails keep showing up in the app.

[YourWishes]

Looking for a solution also. Perhaps related to my use case of running a docker image with both PSQL and redis connected?

[c0r3dump3d]

I also have the same problem, in addition to emails it also happens to me with folders, once they are deleted from the imap server from a external client they are not reflected in Nextcloud. NC 28.0.3 and Mail app 3.5.7.

[meichthys]

Looking for a solution also. Perhaps related to my use case of running a docker image with both PSQL and redis connected?

@YourWishes This is likely not an issue with your environment since this has been a longstanding known issue.

[evilphish]

Experiencing the same with mail 3.5.7 and NC 28.0.4. Other clients I use are K9 Mail which is set to immediately delete messages on the server and Thunderbird which is set to expunge on exit. Between K9 and Thunderbird there are no issues, if I delete a mail on one client it disappears on the other. Nextcloud Mail is the only problem child here displaying ghosts of deleted messages in the inbox.

[mayermart]

I'd like to correct myself: Contrary to my previous comment and like @LostinSpacetime already added, disabling Priority E-Mails was no permanent solution.

[AlexanderZuehr]

Any news with this problem, a lot of our users having this problem and I am not able to activate debug mode and delete cache for everyone.

[pedrolucasbp]

I've tested with Roundcube, K9 and Thunderbird. I confirm the @evilphish comment. All IMAP Clients are OK with deleted messages from other clients, only Nextcloud Mail have this issue.

Any news for this problem?

Thank you in advance.

[ChristophWurst]

We suspect it could be an issue with the Horde cache being flushed too late during cron job execution. We'll add explicit flushing when the IMAP connection is closed.

[meichthys]

We suspect it could be an issue with the Horde cache being flushed too late during cron job execution. We'll add explicit flushing when the IMAP connection is closed.

This would seem quite plausible since we can currently enable Debug mode and use the 'Clear Cache' Mailbox option to temporarily fix the issue.

[blacklight]

Any progress with this, release plans or workarounds? This issue has been the single blocker preventing me from adopting NC Mail so far. I use a multitude of mail clients, and every time I open NC Mail I'm faced with the "unremovable ghost messages" issue.

[rikmeijer]

I created a temporary work-around (am using Nextcloud AIO)

/var/lib/postgresql/data/psql (container)

#!/bin/sh

export PGUSER=$POSTGRES_USER
export PGPASSWORD=$POSTGRES_PASSWORD
export PGDATABASE=$POSTGRES_DB

psql "$@"

/var/lib/postgresql/data/psql-clear-messages (container)

#!/bin/sh

/var/lib/postgresql/data/psql -c "DELETE from oc_mail_recipients WHERE message_id IN (SELECT id FROM oc_mail_messages WHERE mailbox_id=$1);"
/var/lib/postgresql/data/psql -c "DELETE FROM oc_mail_messages WHERE mailbox_id=$1;"

On host machine (in .bashrc):

alias occ='sudo docker exec -ti --user www-data nextcloud-aio-nextcloud /var/www/html/occ'
alias nc-clear-mail='sudo docker exec -it nextcloud-aio-database /var/lib/postgresql/data/psql-clear-messages'

Crontab on host machine

@hourly nc-clear-mail <MAILBOX_ID>; occ mail:account:sync <MAIL_ACCOUNT_ID> --force

It is nasty, but it keeps my orphaned mails away

[GVLLIFESTYLE]

When will this be fixed?

[wriver4]

I had a similar issue and went into the MariaDB table and deleted them. I am curious if it has to do with when the user creates a folder and the sync and/or background sync are not automatically checked.