Rework draft handling (on IMAP)

[ChristophWurst]

Current state

Drafts do not work so well right now. Due to IMAP restrictions we recreate the messages for every byte that changes in the editor. Sometimes old version remain, other times you send the final message and there is a large backlog of drafts to be stored before the message is going out. As recently discussed we have to improve this.

We need to do some research about how other web mail clients handle this. Do they also store to IMAP directly? Could we save to the db first and only when the user exits the current message to to IMAP? …

Implementation

This section aims to describe the high level mechanics of the improved drafts handling with working copies.

Principles

1. Outside the boundaries of a transaction no more than one version of a draft exists. The draft is either on IMAP or in the local database.
2. The drafts handling is compatible with other IMAP clients.

Use Case 1 - composing a new message that is sent right away

1. The user opens the Mail app and clicks New message
2. As the user types, pauses, continues, etc, the app periodically saves working copies-> T1+T3
3. The user clicks Send: the frontend saves the last editor state as another draft revision. Then the frontend sends a request to the backend to convert the local draft to a message in the local outbox. -> T5
4. The message is sent by a trigger from the frontend or a background job in the backend like it was done before

Use Case 2 - composing a new message without sending it

1. The user opens the Mail app and clicks New message
2. As the user types, pauses, continues, etc, the app periodically saves working copies -> T1+T3
3. The user closes the composer modal: the frontend saves the last editor state as another draft revision. Then the frontend sends a request to the backend to move the local draft to a draft message on IMAP. The move operation deletes the local draft from the database. -> T6

Use Case 3 - composing a new message but closing the browser

1. The user opens the Mail app and clicks New message
2. As the user types, pauses, continues, etc, the app periodically saves working copies -> T1+T3
3. The user closes the browser
4. A background job in the backend finds the stale local draft: the backend moves the local draft to a draft message on IMAP. The move operation deletes the local draft from the database. A threshold between current time and the draft last revision timestamp is the indicator for the staleness of a local draft. -> T6

Use Case 4 - composing a new message but discarding it

1. The user opens the Mail app and clicks New message
2. As the user types, pauses, continues, etc, the app periodically saves working copies -> T1+T3
3. The user clicks the draft delete icon: the frontend sends a request to the backend to delete the local draft. -> T4

Use Case 5 - open a draft but make no edits

1. The user opens the Mail app, navigates to the drafts mailbox and opens one of the messages
2. The user closes the composer modal

Use Case 6 - resuming a draft

1. The user opens the Mail app, navigates to the drafts mailbox and opens one of the messages
2. *The user edits the draft in the composer -> T2+T3
3. As the user types, pauses, continues, etc, the app periodically saves working copies-> T3
4. Continue UC1-3 depending on user action

Transaction 1 - create local draft for new message

1. Frontend sends draft data to backend
2. Backend stores draft data in database
3. Backend returns draft data including its database id

Transaction 2 - create local draft from IMAP draft

1. Frontend sends draft data and/or draft message ID to backend. The id is the id of the database cache, not the IMAP UID
2. Backend stores draft data in database
3. Backend deletes draft message from IMAP
4. Backend returns draft data including its database id

Transaction 3 - create revision of local draft

1. Frontend sends draft id and data to the backend
2. Backend replaces data of existing record in database.
3. Backend returns the updated draft including its unchanged database id

Transaction 4 - delete local draft

1. Frontend sends draft id to the backend
2. Backend deletes draft from database

Transaction 5 - send local draft

1. Frontend sends draft id to the backend
2. Backend converts local draft to local outbox message

Transaction 6 - create IMAP draft from local draft

1. Frotnend sends draft id to the backend
2. Backend creates an IMAP message from the draft data in the user's drafts mailbox
3. Backend deletes local draft from database

Prioritization

1 Backend changes

• [ ] #7076

2 Frontend changes

• [ ] #7077 cc @kesselb @LukasReschke @miaulalala @StCyr

[StCyr]

‘Found this: https://developers.google.com/gmail/api/guides/drafts

Are they talking about the same root problem as the one creating all these troubles here?

[miaulalala]

Thunderbirds saving drafts is somewhere around here: https://hg.mozilla.org/comm-central/file/tip/mail/components/compose/content/MsgComposeCommands.js#l5772

[ChristophWurst]

‘Found this: https://developers.google.com/gmail/api/guides/drafts

Are they talking about the same root problem as the one creating all these troubles here?

This actually reads like what we currently do already.

Thunderbirds

What I noticed while testing Thunderbird is that they do not have auto save and therefore potentially fewer versions of a message, mostly depending on how often the user saved manually.

[kesselb]

Rainloop does not auto save. To save a draft one has to click save.

[StCyr]

Rainloop does not auto save. To save a draft one has to click save.

Sure that saving every few key strokes is not necessary. Saving, let’s say, every 30 seconds seems more than enough imho

[ChristophWurst]

Sure that saving every few key strokes is not necessary. Saving, let’s say, every 30 seconds seems more than enough imho

Right. Currently we do it every 700ms. That is a lot. Let's bump it to 10s and see how that performs. Too long has to risk that when the browser crashes or similar we don't always save the latest version.

[miaulalala]

Summary of an internal discussion about drafts:

Solution 1: don't have drafts on IMAP at all Solution 2: let somebody else figure it out Solution 3: Save drafts locally and only send them upstream after a certain amount of time (related: https://github.com/nextcloud/mail/issues/5438) Solution 3a: let the user save a draft to imap via a button Solution 4: Don't sync to IMAP, only store them locally.

[LostinSpacetime]

We are using the mail app in our school since last week and I got already a couple of unhappy teachers that lost their emails because of #5518. A safe and probably easy fix would be a "Save as draft"-button.. maybe combined with a fairly long autosync time (~1 minute).

[ChristophWurst]

Provide API endpoints for the frontend - GET, POST, PUT, DELETE - should already have lots of code there from the outbox

Simplification idea:

• I don't think we need GET. The IMAP drafts will be read through the mailbox listing as before.
• You can use PUT (idempotent) to update a draft and POST to move it back to IMAP.
• Additionally we can use DELETE to discard a local draft and its IMAP associate, if any.
• Lastly there should be the route to move a local draft to the outbox. At that point we delete the IMAP draft, if any.

[jf34000]

Hello, We use nextcloud mail app in a small organization since 1 week, and we already have thousands drafts in "Draft" mailbox. Did someone find a workaround ? For example a way to disable automatic drafts ? Cause this issue is known since so many months... could stay as it for years it seems :o(

Please help ! Cause we are going to rent 10Go more just for mail drafts :o)

[ChristophWurst]

This is in progress.

[ChristophWurst]

Label dropped because it was a duplicate of the attribute at https://github.com/orgs/nextcloud/projects/61. The priority does not change.

[trendzetter]

Is there any plans to make this app production ready? I tried this like years ago and it was in the same unusable state it is now For now I can only advise to disable the Nextcloud Mail app and use the externally developed Rainloop app.

We use nextcloud mail app in a small organization since 1 week, and we already have thousands drafts in "Draft" mailbox. Did someone find a workaround ? For example a way to disable automatic drafts ? Cause this issue is known since so many months... could stay as it for years it seems :o(

Please help ! Cause we are going to rent 10Go more just for mail drafts :o)

[ChristophWurst]

This task is in progress.

[ChristophWurst]

Use Case 5 - open a draft but make no edits

Works with the help of https://github.com/nextcloud/mail/pull/7661

[neeraj-nandan]

Can someone please let me know when will this #4768 issue be fixed, resolved and updated in the app?

By when can we expect this issue to be resolved?

Please provide a date. It would be highly regarded.

[ChristophWurst]

When https://github.com/nextcloud/mail/issues/7077 is done.