Mailspring [WIP] Customizable notification options per account

Adds a new dropdown in the account settings to allow notifications to be configured for the account:

The options are:

Inbox Only: Current behaviour of only showing notifications for emails in the inbox. This is the default
All Folders: Shows notifications for all folders. Technically this isn't actually "all" as it excludes spam and trash, but it's what users would expect "all" to mean
None: Completely disables notifications of new emails for the account

I'd like to implement per-folder notification settings, but that can come as a followup

This PR solves the following feature requests:

This is still a WIP as:

I'm hitting a bug where new emails in folders don't appear until a long time later, even after I do "Sync New Mail Now". Inbox is fine, but folders aren't. I tried waiting ten minutes and doing a sync, and the new emails still didn't appear. However, they do appear after some period of time (unsure as to how long it takes) or when I restartg Mailspring. This is blocking me from fully testing. Related bug report: https://community.getmailspring.com/t/folders-not-updating-until-much-later/943
I want to write a unit test for my shouldNotifyFor function but I can't figure out how to run the tests in this project - npm test just pops up a white screen and throws this error:

index.js:26 Error: Could not find a file at path '../../static/jasmine'
    at ThemeManager.requireStylesheet (file:///C:/src/Mailspring/app/src/theme-manager.ts:163:13)
    at SpecRunner._setupAppEnv (file:///C:/src/Mailspring/app/spec/spec-runner/spec-runner.ts:118:19)
    at SpecRunner.runSpecs (file:///C:/src/Mailspring/app/spec/spec-runner/spec-runner.ts:30:10)
    at Object.<anonymous> (file:///C:/src/Mailspring/app/spec/spec-runner/spec-bootstrap.ts:15:34)
    at Object.<anonymous> (file:///C:/src/Mailspring/app/spec/spec-runner/spec-bootstrap.ts:15:57)
    at Module._compile (internal/modules/cjs/loader.js:968:30)
    at Object.value [as .ts] (C:\src\Mailspring\app\src\compile-cache.js:145:21)
    at Module.load (internal/modules/cjs/loader.js:816:32)
    at Module._load (internal/modules/cjs/loader.js:728:14)
    at Function.Module._load (electron/js2c/asar.js:748:26)
    at Module.require (internal/modules/cjs/loader.js:853:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at setupWindow (file:///C:/src/Mailspring/app/static/index.js:47:3)
    at window.onload (file:///C:/src/Mailspring/app/static/index.js:75:5)

I'm used to Jest tests, but this seems like it uses older Jasmine tests.

Mar 19 '21 22:03 Daniel15

This pull request has been mentioned on Mailspring Community. There might be relevant details there:

https://community.getmailspring.com/t/notifications-on-non-inbox-folders/323/19

Mar 19 '21 22:03 foundry376-bot

Wow. This looks great.

For the unit tests: It seems that you don't have Jasmine - the test runner - available on the system. Did you install all the Dev dependencies?

I will try to give the whole thing a closer look and see if I can give you a pointer on the other issue.

Mar 20 '21 08:03 Phylu

I'm hitting a bug where new emails in folders don't appear until a long time later, even after I do "Sync New Mail Now". Inbox is fine, but folders aren't. I tried waiting ten minutes and doing a sync, and the new emails still didn't appear. However, they do appear after some period of time (unsure as to how long it takes) or when I restartg Mailspring. This is blocking me from fully testing. Related bug report: https://community.getmailspring.com/t/folders-not-updating-until-much-later/943

So this is not related to your notifications, but an issue, that the folders are not synced directly? Unfortunately, I am not sure how the SyncEngine works here.

Mar 20 '21 12:03 Phylu

Did you install all the Dev dependencies?

Yeah, I think they're all there. I ran npm install.

So this is not related to your notifications, but an issue, that the folders are not synced directly?

That's right.

My guess is that Mailspring is only using IMAP IDLE for the inbox (as IDLE only works on one folder at a time), and it's not properly handling subfolders. Usually an email client will use IDLE for all folders (one connection per folder), or at least allow the user to select which folders to get real-time updates for. But even if real-time updates aren't working, I'm not sure why "Sync New Mail Now" isn't working for me (as that should force refresh the folders, right?), and I don't really know much C or C++ so it's hard for me to debug the sync code. @bengotow any ideas?

Mar 20 '21 18:03 Daniel15

Hey folks! Ahh @Daniel15 this looks really cool - I think you're right that there may be a blocker or two on the mailsync side. These days a lot email providers (Gmail, Yahoo, etc) put limits on the number of concurrent IMAP connections each account can have. I think for Gmail it's about 12, but you might be using two for your desktop client, one more for a mobile app, another for the server component of the mobile app, etc.

Currently Mailspring idles on "All Mail" for Gmail accounts which means notifications / new mail arrive in realtime in all your labels, etc. unless you have mail rules set up to "Skip Inbox". For other email providers that use folders, we open one connection and IDLE on the inbox, and use a second connection to load emails you click on / apply changes, etc. I think we could open one connection per folder, but you could hit provider limits pretty quickly and things get "whacky" when you're competing with other email apps connected to the user's account for connections.

For folders we're not IDLEing on, Mailspring wakes every 120s and checks for mail. (main.cpp:218) so it should be notifying in less than 2 minutes. There is an edge case though - if a provider doesn't support the IMAP CONDSTORE and XYZRESYNC extensions, we can't see flag/folder changes on existing emails easily and have to actually scan the contents of the folder again and again, so we only do that every 5-10 minutes I think, even if you say "Sync Mail" over and over. That should only impact the speed we detect messages being marked as read / starred, though and not new mail arriving.

Curious if you're seeing it slower than that? It might be interesting to look at the mailsync-****.log files in your Mailspring data directory and see if anything weird is going on - it's possible that for some reason, syncing your mailbox is taking a really long time and making the "every two minutes loop" slower...

Thanks for your work on this, seems like a great addition!

Mar 26 '21 16:03 bengotow

Thanks for the reply @bengotow :)

I think we could open one connection per folder, but you could hit provider limits pretty quickly

How some other email clients solve it is they allow you to select which folders to get push notifications for. For example, this is what FairEmail (open-source email client for Android) does. There's a few important folders that I want immediate notifications for (for example, I filter all my server downtime alerts into a "servers" folder), but it's fine for the rest of them to be a bit behind.

For folders we're not IDLEing on, Mailspring wakes every 120s and checks for mail. (main.cpp:218) so it should be notifying in less than 2 minutes.

Is that the case even if I click "Sync New Mail Now"? What does "Sync New Mail Now" actually do?

Curious if you're seeing it slower than that?

Yeah I left it for maybe 10 minutes and still didn't get the new emails in subfolders. Restarted Mailspring and they arrived right away. I'll try to get some more logs over the weekend.

it's possible that for some reason, syncing your mailbox is taking a really long time and making the "every two minutes loop" slower...

This is a brand new account I made just for testing Mailspring, so it should be very fast to check. The mail server is in Los Angeles and I'm in the SF Bay Area so there shouldn't be any latency issues.

if a provider doesn't support the IMAP CONDSTORE and XYZRESYNC extensions

I'm using Dovecot as a server. I've got CONDSTORE and QRESYNC:

01 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE LITERAL+ NOTIFY METADATA SPECIAL-USE COMPRESS=DEFLATE QUOTA ACL RIGHTS=texk] Logged in

What is "XYZRESYNC"? I tried searching Google and there was literally only one result - The Mailspring-sync repo

Mar 26 '21 16:03 Daniel15

@bengotow

It might be interesting to look at the mailsync-****.log files in your Mailspring data directory and see if anything weird is going on

I just tried it again today and am still having issues. Here's the log from an initial check:

14704 [2021-03-27 09:57:34.659] [background] [info] Marking all folders as `busy`
14704 [2021-03-27 09:57:34.686] [foreground] [info] Idle exited with code 0
14704 [2021-03-27 09:57:34.716] [foreground] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
14704 [2021-03-27 09:57:34.717] [foreground] [info] Idling on folder INBOX
14704 [2021-03-27 09:57:34.833] [background] [warning] [SLOW] Transaction=markAllFoldersBusy > 80ms (173ms, 0 waiting to aquire)
14704 [2021-03-27 09:57:34.833] [background] [info] Syncing folder list...
14704 [2021-03-27 09:57:34.878] [background] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
14704 [2021-03-27 09:57:34.903] [background] [info] syncFolderChangesViaCondstore - Sent: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:34.937] [background] [info] syncFolderChangesViaCondstore - Drafts: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:34.964] [background] [info] syncFolderChangesViaCondstore - Archive: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:34.995] [background] [info] syncFolderChangesViaCondstore - Trash: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:35.013] [background] [info] syncFolderChangesViaCondstore - Junk: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:35.041] [background] [info] syncFolderChangesViaCondstore - OneExample: modseq 12 to 12, uidnext 11 to 11
14704 [2021-03-27 09:57:35.073] [background] [info] syncFolderChangesViaCondstore - Mailspring: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:35.104] [background] [info] syncFolderChangesViaCondstore - Mailspring/Snoozed: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 09:57:35.105] [background] [info] Sync loop deleting unlinked messages with phase 1.
14704 [2021-03-27 09:57:35.105] [background] [info] Sync loop complete.
14704 [2021-03-27 09:57:35.679] [metadataExpiration] [info] Scanning for expired metadata
14704 [2021-03-27 09:57:35.679] [metadataExpiration] [info] -- Will wake for next expiration in 7200sec

This is the folder I'm testing with:

14704 [2021-03-27 09:57:35.041] [background] [info] syncFolderChangesViaCondstore - OneExample: modseq 12 to 12, uidnext 11 to 11

I sent an email to this account immediately after that sync, into the "OneExample" folder.

On the next sync, Mailspring did not notice the folder had changed. Still "modseq 12 to 12":

14704 [2021-03-27 10:00:13.248] [background] [info] Syncing folder list...
14704 [2021-03-27 10:00:13.249] [foreground] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
14704 [2021-03-27 10:00:13.249] [foreground] [info] Idling on folder INBOX
14704 [2021-03-27 10:00:13.290] [background] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
14704 [2021-03-27 10:00:13.315] [background] [info] syncFolderChangesViaCondstore - Sent: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.340] [background] [info] syncFolderChangesViaCondstore - Drafts: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.370] [background] [info] syncFolderChangesViaCondstore - Archive: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.408] [background] [info] syncFolderChangesViaCondstore - Trash: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.438] [background] [info] syncFolderChangesViaCondstore - Junk: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.464] [background] [info] syncFolderChangesViaCondstore - OneExample: modseq 12 to 12, uidnext 11 to 11
14704 [2021-03-27 10:00:13.485] [background] [info] syncFolderChangesViaCondstore - Mailspring: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.513] [background] [info] syncFolderChangesViaCondstore - Mailspring/Snoozed: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:00:13.514] [background] [info] Sync loop deleting unlinked messages with phase 1.
14704 [2021-03-27 10:00:13.514] [background] [info] Sync loop complete.
14704 [2021-03-27 10:00:14.222] [metadataExpiration] [info] Scanning for expired metadata
14704 [2021-03-27 10:00:14.222] [metadataExpiration] [info] -- Will wake for next expiration in 7200sec

When I clicked "Sync New Mail Now", the same thing:

14704 [2021-03-27 10:01:17.580] [main] [info] Waking all workers...
14704 [2021-03-27 10:01:17.580] [background] [info] Marking all folders as `busy`
14704 [2021-03-27 10:01:17.582] [background] [info] Syncing folder list...
14704 [2021-03-27 10:01:17.599] [foreground] [info] Idle exited with code 0
14704 [2021-03-27 10:01:17.625] [foreground] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
14704 [2021-03-27 10:01:17.625] [foreground] [info] Idling on folder INBOX
14704 [2021-03-27 10:01:17.638] [background] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
14704 [2021-03-27 10:01:17.666] [background] [info] syncFolderChangesViaCondstore - Sent: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.687] [background] [info] syncFolderChangesViaCondstore - Drafts: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.711] [background] [info] syncFolderChangesViaCondstore - Archive: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.735] [background] [info] syncFolderChangesViaCondstore - Trash: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.758] [background] [info] syncFolderChangesViaCondstore - Junk: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.789] [background] [info] syncFolderChangesViaCondstore - OneExample: modseq 12 to 12, uidnext 11 to 11
14704 [2021-03-27 10:01:17.814] [background] [info] syncFolderChangesViaCondstore - Mailspring: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.838] [background] [info] syncFolderChangesViaCondstore - Mailspring/Snoozed: modseq 1 to 1, uidnext 1 to 1
14704 [2021-03-27 10:01:17.838] [background] [info] Sync loop deleting unlinked messages with phase 2.
14704 [2021-03-27 10:01:17.838] [background] [info] Sync loop complete.
14704 [2021-03-27 10:01:18.602] [metadataExpiration] [info] Scanning for expired metadata
14704 [2021-03-27 10:01:18.602] [metadataExpiration] [info] -- Will wake for next expiration in 7200sec

However, after exiting and restarting Mailspring, it worked fine:

23336 [2021-03-27 10:02:21.970] [main] [info] ------------- Starting Sync ([email protected]) ---------------
23336 [2021-03-27 10:02:22.125] [metadata] [info] Metadata delta stream starting...
23336 [2021-03-27 10:02:24.066] [background] [info] Marking all folders as `busy`
23336 [2021-03-27 10:02:24.116] [background] [info] Syncing folder list...
23336 [2021-03-27 10:02:24.493] [background] [info] Syncing folder list...
23336 [2021-03-27 10:02:24.541] [background] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
23336 [2021-03-27 10:02:24.560] [background] [info] syncFolderChangesViaCondstore - Sent: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.590] [background] [info] syncFolderChangesViaCondstore - Drafts: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.615] [background] [info] syncFolderChangesViaCondstore - Archive: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.647] [background] [info] syncFolderChangesViaCondstore - Trash: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.705] [background] [info] syncFolderChangesViaCondstore - Junk: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.729] [background] [info] syncFolderChangesViaCondstore - OneExample: modseq 12 to 13, uidnext 11 to 12
23336 [2021-03-27 10:02:24.778] [background] [info] syncFolderChangesViaCondstore - Changes since HMODSEQ 12: 1 changed, 0 vanished
23336 [2021-03-27 10:02:24.834] [background] [info] syncFolderChangesViaCondstore - Mailspring: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.865] [background] [info] syncFolderChangesViaCondstore - Mailspring/Snoozed: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.866] [background] [info] Sync loop deleting unlinked messages with phase 2.
23336 [2021-03-27 10:02:24.866] [background] [info] Sync loop complete.
23336 [2021-03-27 10:02:24.866] [background] [info] Syncing folder list...
23336 [2021-03-27 10:02:24.922] [background] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
23336 [2021-03-27 10:02:24.934] [foreground] [info] syncFolderChangesViaCondstore - INBOX: modseq 11 to 11, uidnext 6 to 6
23336 [2021-03-27 10:02:24.935] [foreground] [info] Idling on folder INBOX
23336 [2021-03-27 10:02:24.950] [background] [info] syncFolderChangesViaCondstore - Sent: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:24.973] [background] [info] syncFolderChangesViaCondstore - Drafts: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:25.003] [background] [info] syncFolderChangesViaCondstore - Archive: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:25.025] [background] [info] syncFolderChangesViaCondstore - Trash: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:25.051] [background] [info] syncFolderChangesViaCondstore - Junk: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:25.079] [background] [info] syncFolderChangesViaCondstore - OneExample: modseq 13 to 13, uidnext 12 to 12
23336 [2021-03-27 10:02:25.112] [background] [info] syncFolderChangesViaCondstore - Mailspring: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:25.134] [background] [info] syncFolderChangesViaCondstore - Mailspring/Snoozed: modseq 1 to 1, uidnext 1 to 1
23336 [2021-03-27 10:02:25.135] [background] [info] Sync loop deleting unlinked messages with phase 1.
23336 [2021-03-27 10:02:25.135] [background] [info] Sync loop complete.
23336 [2021-03-27 10:02:37.209] [metadataExpiration] [info] Scanning for expired metadata
23336 [2021-03-27 10:02:37.209] [metadataExpiration] [info] -- Will wake for next expiration in 7200sec

This time it said "OneExample: modseq 12 to 13, uidnext 11 to 12" and "Changes since HMODSEQ 12: 1 changed, 0 vanished"

I can message you with the login details for a test account if you'd like? The server-side filters just filter any emails with "One" in the subject line into the "OneExample" folder.

Mar 27 '21 17:03 Daniel15

Any news for this feat ? Need some help maybe ?

Nov 23 '21 11:11 bouteillerAlan

@bouteillerAlan If you want to continue working on this, I would say: Feel free to give it a shot.

Dec 05 '21 16:12 Phylu

Feel free to take over... I had issues with the development environment and with Mailspring not actually fetching new emails in subfolders, so I gave up on this and switched back to Thunderbird.

Dec 05 '21 22:12 Daniel15

As this PR is abandonned, I am closing it for now. If anyone wants to take this up in the future let me know.

Oct 07 '23 16:10 Phylu

Mailspring Mailspring copied to clipboard

[WIP] Customizable notification options per account

Mailspring
Mailspring copied to clipboard