Tasks Sync Failed: database error while applying remote entries

[devvybum]

Have been using Etesync on Android 11 together with Tasks.org for about a week now. Worked fine for the first few days but then started having issues with previously "marked completed" tasks re-appearing in the tasks list. Didn't think much of it at first and just ignored it, but soon after started receiving an Etesync notification that tasks sync has failed (exact wording as per ticket title above). Now the task sync error pops up every minute or so, even after disabling android's account sync settings for Etesync, as well as setting the sync schedule to manual in the Etesync app.

Have tried reinstalling Etesync and Tasks.org, as well as deleting then re-adding my Etesync account to the Etesync app. No dice.

Contacts and calendar sync have been working just fine, only tasks have been problematic.

Below is the debug log.

--- BEGIN DEBUG INFO --- SYNCHRONIZATION INFO Synchronization phase: applying remote entries Account name: USERNAME_REDACTED Authority: org.tasks.opentasks Debug activity source: class com.etesync.syncadapter.syncadapter.SyncNotification$NotificationHandlerActivity (Kotlin reflection is not available)

EXCEPTION: at.bitfire.ical4android.CalendarStorageException: Couldn't apply batch operation at at.bitfire.ical4android.BatchOperation.commit(BatchOperation.kt:54) at at.bitfire.ical4android.AndroidTask.add(AndroidTask.kt:320) at com.etesync.syncadapter.syncadapter.TasksSyncManager.processTask(TasksSyncManager.kt:137) at com.etesync.syncadapter.syncadapter.TasksSyncManager.processItem(TasksSyncManager.kt:89) at com.etesync.syncadapter.syncadapter.SyncManager.applyRemoteItems(SyncManager.kt:470) at com.etesync.syncadapter.syncadapter.SyncManager.performSync(SyncManager.kt:252) at com.etesync.syncadapter.syncadapter.TasksSyncAdapterService$TasksSyncAdapter.onPerformSyncDo(TasksSyncAdapterService.kt:70) at com.etesync.syncadapter.syncadapter.SyncAdapterService$SyncAdapter.onPerformSync(SyncAdapterService.kt:117) at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:334) Caused by: android.content.OperationApplicationException: Too many content provider operations between yield points. The maximum number of operations per yield point is 500 at android.database.DatabaseUtils.readExceptionWithOperationApplicationExceptionFromParcel(DatabaseUtils.java:163) at android.content.ContentProviderProxy.applyBatch(ContentProviderNative.java:599) at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:576) at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:564) at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:80) at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:94) at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:94) at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:94) at at.bitfire.ical4android.BatchOperation.commit(BatchOperation.kt:44) ... 8 more

SOFTWARE INFORMATION EteSync version: 2.2.4 (20204) Installed from: com.android.vending CONFIGURATION Power saving disabled: yes android.permission.READ_CONTACTS permission: granted android.permission.WRITE_CONTACTS permission: granted android.permission.READ_CALENDAR permission: granted android.permission.WRITE_CALENDAR permission: granted org.dmfs.permission.READ_TASKS permission: denied org.dmfs.permission.WRITE_TASKS permission: denied org.tasks.permission.READ_TASKS permission: granted org.tasks.permission.WRITE_TASKS permission: granted System-wide synchronization: automatically Account: USERNAME_REDACTED Address book sync. interval: 240 min Calendar sync. interval: 240 min OpenTasks sync. interval: — Tasks.org sync. interval: — WiFi only: false [CardDAV] Contact group method: GROUP_VCARDS Manage calendar colors: true Address book account: REDACTED (USERNAME_REDACTED kT) Main account: Account {name=USERNAME_REDACTED, type=com.etesync.syncadapter} URL: REDACTED Sync automatically: true

SQLITE DUMP android_metadata | locale | | en_US |

settings | setting | value |

SERVICES DUMP

JOURNALS DUMP

SYSTEM INFORMATION Android version: 11 (unknown) Device: Unknown unknown (unknown)

--- END DEBUG INFO ---

[devvybum]

Managed to "solve" the problem by completely resetting my Etesync account and reinstalling Etesync and Tasks.org from scratch. Not really a proper solution but apart from spending hours experimenting around with the app installs, sync settings, permissions and trying to understand the issue off of the logs, I unfortunately didn't have the technical expertise to troubleshoot deeper than that.

One other thing I did after resetting the account was manually cancelling all recurring tasks as I was getting multiple copies of my recurring tasks appearing as part of the pre-reset issue. I suspect (and I'm sorry for not being able to be more technically-specific about this) that the error had something to do with recurring tasks being borked. The problem hasn't come back so far, so fingers crossed all stays normal.

[eternaltyro]

This error hit me and it's debilitating. The notification is just so incessant that I had to switch off sync-errors in the notification settings. I also think this is affecting tasks app's ability to sync and is causing data loss. Is there any workaround for this? What exactly is causing it?

--- BEGIN DEBUG INFO ---
SYNCHRONIZATION INFO
Synchronization phase: applying remote entries
Account name: USER
Authority: org.tasks.opentasks
Debug activity source: class com.etesync.syncadapter.syncadapter.SyncNotification$NotificationHandlerActivity (Kotlin reflection is not available)

EXCEPTION:
at.bitfire.ical4android.CalendarStorageException: Couldn't apply batch operation
	at at.bitfire.ical4android.BatchOperation.commit(BatchOperation.kt:54)
	at at.bitfire.ical4android.AndroidTask.add(AndroidTask.kt:320)
	at com.etesync.syncadapter.syncadapter.TasksSyncManager.processTask(TasksSyncManager.kt:137)
	at com.etesync.syncadapter.syncadapter.TasksSyncManager.processItem(TasksSyncManager.kt:89)
	at com.etesync.syncadapter.syncadapter.SyncManager.applyRemoteItems(SyncManager.kt:470)
	at com.etesync.syncadapter.syncadapter.SyncManager.performSync(SyncManager.kt:252)
	at com.etesync.syncadapter.syncadapter.TasksSyncAdapterService$TasksSyncAdapter.onPerformSyncDo(TasksSyncAdapterService.kt:70)
	at com.etesync.syncadapter.syncadapter.SyncAdapterService$SyncAdapter.onPerformSync(SyncAdapterService.kt:117)
	at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:334)
Caused by: android.content.OperationApplicationException: Too many content provider operations between yield points. The maximum number of operations per yield point is 500
	at android.database.DatabaseUtils.readExceptionWithOperationApplicationExceptionFromParcel(DatabaseUtils.java:163)
	at android.content.ContentProviderProxy.applyBatch(ContentProviderNative.java:599)
	at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:576)
	at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:564)
	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:80)
	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:94)
	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:94)
	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:94)
	at at.bitfire.ical4android.BatchOperation.commit(BatchOperation.kt:44)
	... 8 more

SOFTWARE INFORMATION
EteSync version: 2.2.4 (20204) 
Installed from: com.google.android.packageinstaller
CONFIGURATION
Power saving disabled: no
android.permission.READ_CONTACTS permission: granted
android.permission.WRITE_CONTACTS permission: granted
android.permission.READ_CALENDAR permission: granted
android.permission.WRITE_CALENDAR permission: granted
org.dmfs.permission.READ_TASKS permission: denied
org.dmfs.permission.WRITE_TASKS permission: denied
org.tasks.permission.READ_TASKS permission: granted
org.tasks.permission.WRITE_TASKS permission: granted
System-wide synchronization: automatically
Account: USER
  Address book sync. interval: 240 min
  Calendar     sync. interval: 240 min
  OpenTasks    sync. interval: —
  Tasks.org    sync. interval: 240 min
  WiFi only: false
  [CardDAV] Contact group method: GROUP_VCARDS
           Manage calendar colors: true
Address book account: NAME (USER oY)
  Main account: Account {name=USER, type=com.etesync.syncadapter}
  URL: XXX
  Sync automatically: true

SQLITE DUMP
android_metadata
	|  locale |
	|  en_IN |
----------
settings
	|  setting | value |
----------

SERVICES DUMP

JOURNALS DUMP

SYSTEM INFORMATION
Android version: 9 ()
Device: Motorola moto g6 ()

--- END DEBUG INFO ---

[marlluslustosa]

I have the same problem. This seems to have to do with the android version. Similar to what is happening recently in Joplin sync: https://github.com/laurent22/joplin/issues/5779.

[tasn]

@marlluslustosa, any idea for resolution?

[marlluslustosa]

@marlluslustosa, any idea for resolution?

I saw that from android 11 (API 30) a lot has changed regarding the permission level of apps to the sdcard and folders in the system https://developer.android.com/about/versions/11/privacy/storage

I found this post that explains well some ways to get around this, on the dev side. https://stackoverflow.com/a/68141908

[marlluslustosa]

@tasn I think I made a mistake in the previous answer. I set the "android:legacy_storage allow" to etesync via ADB (as per here) but it didn't work. So looking deeper into my logs, I noticed the same error common to users above. "android.content.OperationApplicationException: Too many content provider operations between yield points. The maximum number of operations per yield point is 500"

So looking a little further I landed on this page: https://stackoverflow.com/questions/15525440/bulk-update-of-more-than-500-contacts

Just for the record, I have over 3,000 contacts registered on my etesync account. And I think this probably has to do with that 500 contact limit.

[tasn]

Oh, that's interesting. I think we actually process much less at a time, but I wonder if maybe we can limit further.

[lokesh-krishna]

Managed to "solve" the problem by completely resetting my Etesync account and reinstalling Etesync and Tasks.org from scratch.

Can anyone explain how one goes about resetting their account when self-hosting? The account dashboard section is inaccessible for me.

I would love an actual solution but the problem is persistent enough that I would like to try this workaround.

[Encrypt0r]

I had the same issue, and tried to debug it. Unfortunately I could not build the EteSync Android application due to some missing models (com.etesync.syncadapter.model.ServiceEntity), so I ended up in collecting logs, and finding a way to only reset the broken list.

How to get it work again, without resetting everything:

1. Find out which task list is causing the issue. I suggest to do some modifications in all of them, and for the working ones, you will get a notification which lists were successfully synced. The missing one is the broken one.
2. In Tasks, select all the tasks from this broken list, and move them to an empty local list
3. In EteSync delete this collection
4. In EteSync create a new collection
5. In Tasks, move your tasks back from the local list to this new EteSync list Note: You will lose all the history of those tasks, but at least you can keep the latest version from your device.

Some details about the issue:

• For me it started by modifying an existing task. Sporadically it reverted back my modifications in this specific task, to an earlier state. Therefore I suspected this task to cause the problem. I checked the history of it, and in its RAW format, a 'sort' entry appeared 1428 times (X-APPLE-SORT-ORDER:564986952). I don't know whether is it related to the issue, but reverting it back to an earlier state, when there was only one 'sort' entry, did not solve the issue.
• I have also read the debug level logs via LogCat, which was complaining about inserting an item into the database with a non unique fts_propery_id, but this error actually origins from the Tasks application when EteSync is trying to update an entry. I am not even sure whether this issue belongs to: EteSync or Tasks?

Log:

24718-25804/com.etesync.syncadapter I/etesync: [com.etesync.syncadapter.syncadapter.SyncManager] Processing (8/38) UID=c-_PtDvYWJ5vpaobJFaYgi5nYPStPYd6 Etag=kFe6Ec34UhDdQWx4zDTs1A
24718-25804/com.etesync.syncadapter I/etesync: [com.etesync.syncadapter.syncadapter.TasksSyncManager] Updating c-_PtDvYWJ5vpaobJFaYgi5nYPStPYd6 in local calendar
4065-7324/org.tasks D/Profiled: Time spent in Delete: 1 milliseconds
4065-7324/org.tasks D/Profiled: Time spent in Update: 1 milliseconds
4065-7324/org.tasks E/SQLiteDatabase: Error inserting fts_task_id=716 fts_ngram_id=3045 fts_type=4 fts_property_id=269075
    android.database.sqlite.SQLiteConstraintException: UNIQUE constraint failed: FTS_Content.fts_type, FTS_Content.fts_task_id, FTS_Content.fts_property_id (code 2067 SQLITE_CONSTRAINT_UNIQUE)

This insertion in the last line is retried about 20 times.

[abaker]

... which was complaining about inserting an item into the database with a non unique fts_propery_id...

IIRC this comes from the OpenTasks content provider and can be safely ignored

For me it started by modifying an existing task. Sporadically it reverted back my modifications in this specific task, to an earlier state. Therefore I suspected this task to cause the problem. I checked the history of it, and in its RAW format, a 'sort' entry appeared 1428 times (X-APPLE-SORT-ORDER:564986952). I don't know whether is it related to the issue, but reverting it back to an earlier state, when there was only one 'sort' entry, did not solve the issue.

Do any other tasks have more than one sort entry on them? Did this task sync OK once you moved it to a new EteSync collection?

[Encrypt0r]

Do any other tasks have more than one sort entry on them? Did this task sync OK once you moved it to a new EteSync collection?

I have lost this task, as I deleted it from Tasks, and I have cleared the storage of EteSync apk.

I tried to get it back for you from EteSync history: I have rolled back my EteBase server to the broken database, however it seems that the synchronization has stopped since this issue first appeared. The last entry in Change Journal is now from 5/9/2022. Now after this rollback, my tasks after this date were removed from Tasks app, and the task sync issue is back again. I remember that before resetting the storage, I could have seen all the tasks after this date in EteSync apk, and even their revisions. So it seems those changes were not pushed to EteBase server, and they only existed in Tasks and in EteSync apk.

I attached the logcat trace about the issue which is back again, I hope it helps finding the rootcause.

After changing the EteBase database again to the fixed one, and clearing the storage of EteSync, everything works fine again. All my tasks are back.

[Encrypt0r]

I think there are actually 3 different issues here, please help me decide where should I create new issues? for EteSync or Tasks? These are kind of in between:

1. Task Sync Failed
2. Undetected changes
3. X-APPLE-SORT-ORDER

Regarding the 2nd:

• If only the RELATED-TO;RELTYPE=PARENT value is modified, EteSync does not detect any change.
• If I later rename the task, both the new name and the previously changed parent is syncronized to EteSync.
• If I do both at the same time that is correct.

Regarding the 3rd:

Do any other tasks have more than one sort entry on them? Did this task sync OK once you moved it to a new EteSync collection?

Yes, X-APPLE-SORT-ORDER is back again with 2 new tasks, these tasks are again not modifiable in some manner. The tasks now have an unremovable tag.

• If I try to remove the tag, the tag simply returns after a sync, and there is no change detected in EteSync (not even a timestamp).
• If I only rename the task, it gets synchronized correctly, but at least one additional SORT entry is added.
• If I both rename and remove the tag, it gets synchronized: Name is updated successfully, but the tag is only removed from EteSync RAW, it will come back in Tasks. An additional SORT entry is added again.

Neither of the 2nd or 3rd issue is the rootcause for the 1st issue, they are not causing "Task Sync Failed" notifications.