Add support for Webhooks to notify the mail app that new mail has arrived

[SebastianKrupinski]

Is your feature request related to a problem? Please describe.

No

Describe the solution you'd like

Some mail server now support webhooks to notify mail clients that new messages have arrived. It would be nice to have the mail server trigger a mail sync when new messages arrive instead of waiting for the mail app to update.

Describe alternatives you've considered

No response

Work packages

• [x] Clarify if we can use implement a standardized endpoint that can be used by Dovecot and Stalwart

Additional context

No response

[SebastianKrupinski]

Dovecot and Stalwart do support push notifications, but are missing some functionality at the moment to make this feature work.

There is a push notification standard called "Internet Message Store Events RFC 5423", the standard defines what events should be accessible and what parameters should be included. BUT the standard does NOT define how the information should be delivered to the consuming service. Therefor there is no standardization on authentication methods or data format (JSON, XML, Other)

Dovecot

Does support the Internet Message Store Events RFC 5423, but does not have built in capabilities to trigger HTTP endpoints on the consuming system. Instead Dovecot uses a plugin structure that triggers methods for each event with the required data according to the RFC. Dovecot does have a pre made plugin that supports Lua scripting which could be used to either directly add changes to the NC database or used to trigger a endpoint on NC.

Pros

• Implements Internet Message Store Events RFC 5423

Cons

• Requires custom plugin/script to transmit notifications

Stalwart

Does have built in capabilities to trigger HTTP endpoints on a consuming system. BUT does not support the Internet Message Store Events RFC 5423 and does not have the ability to use a plugin or scripting to accomplish this. Another issue with Stalwart is the lack of required information in mail operation notification, mainly the accountName (user login or email address) as all operation notification only contain a internal "accountId".

Pros

• Can trigger HTTP endpoints out of the box (JSON)
• Supports all mail operations

Cons

• Does not support RFC required parameters
• Would require revers connections to retrieve required information

[ChristophWurst]

Then let's go with Dovecot + Script for now

[ChristophWurst]

For documentation purposes, @SebastianKrupinski please keep this ticket updated with the blockers you've found with Stalwart. Thanks.

[SebastianKrupinski]

Update:

Dovecot - With the use of a lua script most functions are working properly (Mailbox Create/Rename/Move/Delete), but we are missing some information on the server side to make Message Create/Move/Delete work properly. Waiting for contact information for dovecot

Stalwart - Although Stalwart has webhooks support they do not use the Internet Message Store Events RFC 5423 format. The main issue is missing information in the webhook messages, as all folder references are internal id's "10" and not IMAP folder names "Inbox\Test Folder" due to this there is no way for us to translate this information properly for our use.

[SebastianKrupinski]

Update:

Had a conversation with Dovecot, the information we are looking for in the hook request, is not available. Therefore we still have a situation where MessageAppend (Create/Move/Delete) works but is missing some important information like Date and Preview.

A solution to this would be to:

• make a outbound IMAP call to the server to retrieve the missing information on a per message basis. Down side is if the a user moves 20/30 messages at once, this would cause one connection to the server per message.
• make a outbound IMAP call to the server to sync the whole mailbox. This could be done on time interval, by tracking when the mailbox was last sync'd and only syncing every few minutes, versus on every hook trigger.

All though both these are viable options, they both have a side effect. From my tests, hook calls in Dovecot are blocking, the IMAP response is paused until the hook call returns a status.

Possible ways around this issue:

• Do early status return but continue the processing (not sure if this is possible, at the moment)
• Mark the folder as dirty and schedule a one time background job to sync that folder
• Hijack task processing (if available) with a custom job to run a sync instantly in the back ground (this would be the best solution)

[ChristophWurst]

Thanks for the update. The effort seems rather high right now for a feature that will only be used by a small number of setups. Therefore we will put a hold on the change.