sync-engine
sync-engine copied to clipboard
Published
20 hours ago •
nylas
nylas
Labels not synced properly
trafficstars
Scenario: Gmail account with one thread (3 messages). Messages 1 and 2 are in inbox, message 3 is sent.
Steps to reproduce:
- Remove the inbox label from the thread => all emails are removed from Inbox as expected
- Add the inbox label to the thread => all emails show up in Inbox as expected
- Archive the thread in Gmail => Nylas will never sync the final state
This happens for both the new labels API as well as the old tags API.
Details below.
Scenario:
MSG1_RFC822 = '''Subject: Hello
From: [email protected]
To: [email protected]
Reply-To: [email protected]
Message-Id: <[email protected]>
Date: Wed, 30 Jan 2013 10:00:00 -0800
first message'''.replace('\n', '\r\n')
MSG2_RFC822 = '''Subject: Hello
From: [email protected]
To: [email protected]
Cc: [email protected]
Message-Id: <[email protected]>
References: <[email protected]>
second message'''.replace('\n', '\r\n')
MSG3_RFC822 = '''Subject: Re: Hello
From: [email protected]
To: [email protected]
Message-Id: <[email protected]>
References: <[email protected]>
Date: Sat, 30 Jan 2100 10:00:00 -0800
third message'''.replace('\n', '\r\n')
GMAIL_DATA = {
'INBOX': [
{ 'data': MSG1_RFC822, 'flags': ['\\Seen'] },
{ 'data': MSG2_RFC822, 'flags': [] },
],
'[Gmail]/Drafts': [],
'[Gmail]/Important': [],
'[Gmail]/Sent Mail': [ { 'data': MSG3_RFC822, 'flags': ['\\Seen'] } ],
'[Gmail]/Spam': [],
'[Gmail]/Starred': [],
'[Gmail]/Trash': [],
}
Thread after Nylas' initial sync (all labels set correctly):
% curl 'http://127.0.0.1:5555/n/7n9aakq7b24kn2xn21y7ris4c/threads/1edbkhhn46wp553g61t2j104i?view=expanded'
{
"drafts": [],
"first_message_timestamp": 1439491680,
"has_attachments": false,
"id": "1edbkhhn46wp553g61t2j104i",
"labels": [
{
"display_name": "Sent Mail",
"id": "2ow3wkp19f7zy1i7hma8krnvf",
"name": "sent"
},
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
},
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
}
],
"last_message_received_timestamp": 1439491680,
"last_message_timestamp": 1439491681,
"messages": [
{
"bcc": [],
"cc": [],
"date": 1439491680,
"files": [],
"from": [
{
"email": "[email protected]",
"name": ""
}
],
"id": "4zxh082ycxfho28b7gqrvrpzq",
"labels": [
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
},
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
}
],
"message_id_header": "<[email protected]>",
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "message",
"reply_to": [
{
"email": "[email protected]",
"name": ""
}
],
"snippet": "first message",
"starred": false,
"subject": "Hello",
"thread_id": "1edbkhhn46wp553g61t2j104i",
"to": [
{
"email": "[email protected]",
"name": ""
}
],
"unread": false
},
{
"bcc": [],
"cc": [
{
"email": "[email protected]",
"name": ""
}
],
"date": 1439491680,
"files": [],
"from": [
{
"email": "[email protected]",
"name": ""
}
],
"id": "bd2mto66nti5l5vpm1ice32ym",
"labels": [
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
},
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
}
],
"message_id_header": "<[email protected]>",
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "message",
"reply_to": [],
"snippet": "second message",
"starred": false,
"subject": "Hello",
"thread_id": "1edbkhhn46wp553g61t2j104i",
"to": [
{
"email": "[email protected]",
"name": ""
}
],
"unread": true
},
{
"bcc": [],
"cc": [],
"date": 1439491681,
"files": [],
"from": [
{
"email": "[email protected]",
"name": ""
}
],
"id": "2vvprqnrtxti1pkppmmchneuh",
"labels": [
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
},
{
"display_name": "Sent Mail",
"id": "2ow3wkp19f7zy1i7hma8krnvf",
"name": "sent"
}
],
"message_id_header": "<[email protected]>",
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "message",
"reply_to": [],
"snippet": "third message",
"starred": false,
"subject": "Re: Hello",
"thread_id": "1edbkhhn46wp553g61t2j104i",
"to": [
{
"email": "[email protected]",
"name": ""
}
],
"unread": false
}
],
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "thread",
"participants": [
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
}
],
"snippet": "third message",
"starred": false,
"subject": "Hello",
"tags": [
{
"id": "sent",
"name": "[Gmail]/Sent Mail"
},
{
"id": "inbox",
"name": "Inbox"
},
{
"id": "all",
"name": "[Gmail]/All Mail"
},
{
"id": "unread",
"name": "unread"
}
],
"unread": true,
"version": 1
}
First, we remove the inbox label:
% curl -XPUT 'http://127.0.0.1:5555/n/7n9aakq7b24kn2xn21y7ris4c/threads/1edbkhhn46wp553g61t2j104i?view=expanded' -d '{"labels": [ "2ow3wkp19f7zy1i7hma8krnvf", "2qpw0la3shbr847kohspyyso7" ]}'
{
"draft_ids": [],
"first_message_timestamp": 1439491680,
"has_attachments": false,
"id": "1edbkhhn46wp553g61t2j104i",
"labels": [
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
},
{
"display_name": "Sent Mail",
"id": "2ow3wkp19f7zy1i7hma8krnvf",
"name": "sent"
}
],
"last_message_received_timestamp": 1439491680,
"last_message_timestamp": 1439491681,
"message_ids": [
"4zxh082ycxfho28b7gqrvrpzq",
"bd2mto66nti5l5vpm1ice32ym",
"2vvprqnrtxti1pkppmmchneuh"
],
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "thread",
"participants": [
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
}
],
"snippet": "third message",
"starred": false,
"subject": "Hello",
"tags": [
{
"id": "all",
"name": "[Gmail]/All Mail"
},
{
"id": "sent",
"name": "[Gmail]/Sent Mail"
},
{
"id": "unread",
"name": "unread"
}
],
"unread": true,
"version": 3
}
Then, we re-add the inbox label:
% curl -XPUT 'http://127.0.0.1:5555/n/7n9aakq7b24kn2xn21y7ris4c/threads/1edbkhhn46wp553g61t2j104i?view=expanded' -d '{"labels": [ "2ow3wkp19f7zy1i7hma8krnvf", "2qpw0la3shbr847kohspyyso7", "au1hjpgnyepof0xmkpvleydtv" ]}'
{
"draft_ids": [],
"first_message_timestamp": 1439491680,
"has_attachments": false,
"id": "1edbkhhn46wp553g61t2j104i",
"labels": [
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
},
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
},
{
"display_name": "Sent Mail",
"id": "2ow3wkp19f7zy1i7hma8krnvf",
"name": "sent"
}
],
"last_message_received_timestamp": 1439491680,
"last_message_timestamp": 1439491681,
"message_ids": [
"4zxh082ycxfho28b7gqrvrpzq",
"bd2mto66nti5l5vpm1ice32ym",
"2vvprqnrtxti1pkppmmchneuh"
],
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "thread",
"participants": [
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
}
],
"snippet": "third message",
"starred": false,
"subject": "Hello",
"tags": [
{
"id": "inbox",
"name": "Inbox"
},
{
"id": "all",
"name": "[Gmail]/All Mail"
},
{
"id": "sent",
"name": "[Gmail]/Sent Mail"
},
{
"id": "unread",
"name": "unread"
}
],
"unread": true,
"version": 6
}
Now, we archive the thread in Gmail and wait. The inbox label will never be removed from Nylas. This is the eventual (incorrect) result:
% curl 'http://127.0.0.1:5555/n/7n9aakq7b24kn2xn21y7ris4c/threads/1edbkhhn46wp553g61t2j104i?view=expanded'
{
"drafts": [],
"first_message_timestamp": 1439491680,
"has_attachments": false,
"id": "1edbkhhn46wp553g61t2j104i",
"labels": [
{
"display_name": "Sent Mail",
"id": "2ow3wkp19f7zy1i7hma8krnvf",
"name": "sent"
},
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
},
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
}
],
"last_message_received_timestamp": 1439491680,
"last_message_timestamp": 1439491681,
"messages": [
{
"bcc": [],
"cc": [],
"date": 1439491680,
"files": [],
"from": [
{
"email": "[email protected]",
"name": ""
}
],
"id": "4zxh082ycxfho28b7gqrvrpzq",
"labels": [
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
},
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
}
],
"message_id_header": "<[email protected]>",
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "message",
"reply_to": [
{
"email": "[email protected]",
"name": ""
}
],
"snippet": "first message",
"starred": false,
"subject": "Hello",
"thread_id": "1edbkhhn46wp553g61t2j104i",
"to": [
{
"email": "[email protected]",
"name": ""
}
],
"unread": false
},
{
"bcc": [],
"cc": [
{
"email": "[email protected]",
"name": ""
}
],
"date": 1439491680,
"files": [],
"from": [
{
"email": "[email protected]",
"name": ""
}
],
"id": "bd2mto66nti5l5vpm1ice32ym",
"labels": [
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
},
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
}
],
"message_id_header": "<[email protected]>",
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "message",
"reply_to": [],
"snippet": "second message",
"starred": false,
"subject": "Hello",
"thread_id": "1edbkhhn46wp553g61t2j104i",
"to": [
{
"email": "[email protected]",
"name": ""
}
],
"unread": true
},
{
"bcc": [],
"cc": [],
"date": 1439491681,
"files": [],
"from": [
{
"email": "[email protected]",
"name": ""
}
],
"id": "2vvprqnrtxti1pkppmmchneuh",
"labels": [
{
"display_name": "Inbox",
"id": "au1hjpgnyepof0xmkpvleydtv",
"name": "inbox"
},
{
"display_name": "Sent Mail",
"id": "2ow3wkp19f7zy1i7hma8krnvf",
"name": "sent"
},
{
"display_name": "All Mail",
"id": "2qpw0la3shbr847kohspyyso7",
"name": "all"
}
],
"message_id_header": "<[email protected]>",
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "message",
"reply_to": [],
"snippet": "third message",
"starred": false,
"subject": "Re: Hello",
"thread_id": "1edbkhhn46wp553g61t2j104i",
"to": [
{
"email": "[email protected]",
"name": ""
}
],
"unread": false
}
],
"namespace_id": "7n9aakq7b24kn2xn21y7ris4c",
"object": "thread",
"participants": [
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
},
{
"email": "[email protected]",
"name": ""
}
],
"snippet": "third message",
"starred": false,
"subject": "Hello",
"tags": [
{
"id": "sent",
"name": "[Gmail]/Sent Mail"
},
{
"id": "inbox",
"name": "Inbox"
},
{
"id": "all",
"name": "[Gmail]/All Mail"
},
{
"id": "unread",
"name": "unread"
}
],
"unread": true,
"version": 6
}
Hi Thomas. Thanks for the detailed bug report.
This bug is an unintended consequence of this line: https://github.com/nylas/sync-engine/blob/master/inbox/mailsync/backends/imap/common.py#L228 . It was added because there's a very annoying interaction with the delta stream API. We're still looking into the best way to solve this.
In the meantime, you can try to remove the (now - actionlog.updated_at).seconds >= 90 or remove it altogether if you're not using the delta API.
Karim
Thanks Karim for getting back.
We're using the delta API. What's the downside of removing this line?
Removing it may make deltas look weird. Let’s say you have a thread with two messages:
Thread:
M1:
labels: Inbox, sent
M2:
labels: Inbox
version: 1
If you remove the archive the message using the API, the sync engine will archive each message individually. However, because of the way the sync is architected, there's a potential race condition happening. Here are the transactions we're creating:
-
We apply the label change locally and queue the actions for syncback. Everything is a-okay.
Transaction 1: Thread: labels: sent version: 2 -
The first action executes, the labels are recomputed. Because the second action hasn't run yet, the labels look stale:
Transaction 2: Thread: labels: sent, inbox version: 3 -
The third action executes, and the labels finally look like they're supposed to.
Transaction 3: Thread: labels: sent version: 4
This is a tricky condition and we're trying to find the best way to solve it.
Is this why when I delete/archive Gmail messages on the web or in N1 on another computer, they sometimes still show up in N1 on this computer?
I sometimes use the web client or my phone too, and having a bunch of old emails sitting around in N1 is a drag (it makes me have to remember what I've already archived/deleted, wasting brainpower). Is this still being looked at since the last update in ~October?