twenty icon indicating copy to clipboard operation
twenty copied to clipboard

Create a service to do a partial sync with the gmail API using the historyId

Open bosiraphael opened this issue 1 year ago • 1 comments

Scope & Context

After doing the full sync when creating a connected account, we need to call a cron job regularly to sync all the emails.

Technical inputs

Create a new service in packages/twenty-server/src/workspace/messaging which uses the users.history.list method from the gmail API to get all the changes since the last historyId and update the database.

  • get the lastSyncHistoryId from "messageChannel"
  • call users.history.list with startHistoryId=lastSyncHistoryId, historyTypes=[messageAdded, messageDeleted]
  • we get the following response { "history": [ { object (History) } ], "nextPageToken": string, "historyId": string }
  • If there is a nextPageToken, call users.history.list with pageToken=nextPageToken startHistoryId=lastSyncHistoryId, historyTypes=[messageAdded, messageDeleted]
  • Get all the messagesAdded and messagesDeleted id and threadId
  • Get threads with users.threads.get and messages withusers.messages.get in batch requests
  • Parse the results
  • Store the added messages in the db
  • Delete the deleted messages from the db
  • Store the new historyId in lastSyncHistoryId in the messageChannel
  • Maybe store a timestamp of the last sync

bosiraphael avatar Jan 04 '24 15:01 bosiraphael

  • To check if we want to import SPAM
  • To check what to do with TRASH messages -> If we want them, we need to add historyType for changed labels

Weiko avatar Jan 05 '24 15:01 Weiko