android icon indicating copy to clipboard operation
android copied to clipboard

Published 20 hours ago • verified publisher icon nextcloud

[Meta] Crashes caused by too much data in SQLite rows

Open AlvaroBrey opened this issue 3 years ago • 10 comments

This issue is intended to be a compilation of crashes caused (apparently) by too much data in SQL. We've patched some of these in the past but it's evident that a more integral solution or rework is needed to stop this happening all the time.

Issues

Row too big to fit in CursorWindow

  • https://github.com/nextcloud/android/issues/10199
  • https://github.com/nextcloud/android/issues/10150
  • https://github.com/nextcloud/android/issues/9693
  • https://github.com/nextcloud/android/issues/9384
  • many more closed ones

Couldn't read row... Make sure Cursor is initialized

  • https://github.com/nextcloud/android/issues/10193
  • https://github.com/nextcloud/android/issues/10192
  • https://github.com/nextcloud/android/issues/9948
  • https://github.com/nextcloud/android/issues/6103
  • https://github.com/nextcloud/android/issues/10245
  • https://github.com/nextcloud/android/issues/11399
  • many more closed ones

Causes

The filelist table has several columns that can grow indefinitely, causing this problem:

  • rich_workspace contains, for folders, the entire contents of the README.md file for that folder.
    • This is likely the biggest cause of this crash, as it's very easy to create a README.md file over 2mb in size (by adding a lot of embedded images, for example). good example in #9693.
    • This should just not be stored in a sql database, period.
    • Ideas:
      * store the README as a file in private storage and use this field as a reference to that file * store the first N lines of the README, only fetch the entire one when displaying it. * don't store the README at all, just grab the contents from the README file when displaying the folder.
  • sharees contains a json-encoded list of sharees for the file. This grows indefinitely as more sharees are added.
    • Crash could be possible if a file has tons of sharees. However this should be rather hard to trigger if not on purpose; even a column with 1000 sharees will only occupy around 100 kB
    • This should be stored in a separate table and JOINed when querying, or something like that. Not sure if that is even possible with the current ContentResolver approach.
  • note contains the note added to the share when sharing.
    • Not sure if this can be of arbitrary length, but it's likely not as easy to make grow as only plaintext is possible.

In the past, we've mitigated this issue at some places by using a projection to avoid fetching all columns. However, when retrieving a list of OCFiles, there's no way around this as we actually do want all fields. To really fix this we'll need to change how the fields above are handled.

We could mitigate this further by increasing the cursor size (supported on android>=9, and possible via reflection before that) to something like 50 mB instead, but that will just make the problem less common instead of fixing it.

cc @tobiasKaminsky

AlvaroBrey avatar May 09 '22 07:05 AlvaroBrey

Is this also a duplicate crash?

Cause of error

Exception in thread "main" android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=7538, totalRows=6830
    at android.database.sqlite.SQLiteConnection.nativeExecuteForCursorWindow(Native Method)
    at android.database.sqlite.SQLiteConnection.executeForCursorWindow(SQLiteConnection.java:1003)
    at android.database.sqlite.SQLiteSession.executeForCursorWindow(SQLiteSession.java:838)
    at android.database.sqlite.SQLiteQuery.fillWindow(SQLiteQuery.java:62)
    at android.database.sqlite.SQLiteCursor.fillWindow(SQLiteCursor.java:153)
    at android.database.sqlite.SQLiteCursor.onMove(SQLiteCursor.java:123)
    at android.database.AbstractCursor.moveToPosition(AbstractCursor.java:255)
    at android.database.AbstractCursor.moveToNext(AbstractCursor.java:287)
    at android.database.CursorWrapper.moveToNext(CursorWrapper.java:206)
    at com.owncloud.android.datamodel.FileDataStorageManager.getGalleryItems(FileDataStorageManager.java:2226)
    at com.owncloud.android.datamodel.FileDataStorageManager.getAllGalleryItems(FileDataStorageManager.java:2169)
    at com.owncloud.android.ui.adapter.GalleryAdapter.showAllGalleryItems(GalleryAdapter.kt:149)
    at com.owncloud.android.ui.fragment.GalleryFragment.showAllGalleryItems(GalleryFragment.java:262)
    at com.owncloud.android.ui.asynctasks.GallerySearchTask.onPostExecute(GallerySearchTask.java:127)
    at com.owncloud.android.ui.asynctasks.GallerySearchTask.onPostExecute(GallerySearchTask.java:48)
    at android.os.AsyncTask.finish(AsyncTask.java:771)
    at android.os.AsyncTask.access$900(AsyncTask.java:199)
    at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:788)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loopOnce(Looper.java:201)
    at android.os.Looper.loop(Looper.java:288)
    at android.app.ActivityThread.main(ActivityThread.java:7870)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)

App information

  • ID: com.nextcloud.client
  • Version: 30200390
  • Build flavor: gplay

Device information

  • Brand: google
  • Device: oriole
  • Model: Pixel 6
  • Id: SQ3A.220605.009.B1
  • Product: oriole

Firmware

  • SDK: 32
  • Release: 12
  • Incremental: 8650216

svenb1234 avatar Jun 16 '22 16:06 svenb1234

Is this also a duplicate crash?

I don't think we've seen that particular stacktrace (with getAllGalleryItems) before, but the root cause is likely still the same. Can you reproduce this crash reliably?

AlvaroBrey avatar Jun 17 '22 07:06 AlvaroBrey

Each time I open the media view I get one of the two crashes. Either something with "Couldn't read row x, col 3 from CursorWindow." or " android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=..."

Shall I open a separate report for the latter?

svenb1234 avatar Jun 17 '22 09:06 svenb1234

Shall I open a separate report for the latter?

No need. In reality these are all the same underlying issue, that's why I created this meta issue. The problem is that the filelist database table has text fields that can grow indefinitely, and this can cause rows to be far too big for the SQLite engine to handle them.

AlvaroBrey avatar Jun 17 '22 09:06 AlvaroBrey

@svenb1234 I've added a list of possible causes for this crash at the end of the first post in this issue. Just for sanity check, does any of them look like a probable cause for your scenario?

AlvaroBrey avatar Jun 17 '22 09:06 AlvaroBrey

To be honest, I am not sure whether one of those causes applies to me. I have only three users on the server. admin, user1 and user2. There are several shares, e.g. a folder which contains a lot of pictures in sub-folders. That folder is shared from user1 to user2. What other kind of share would count for sharees?

I do not use the rich workspace. It i disabled in the Webui.

The share does not have any notes. If I add a note to the share it is short.

As decribed in #10389 I get these crashes very often, without any interaction, if new previews were added. Usually there are round about 30+ new pictures, sometimes more. To me it seems that it builds some cache and crashes while doing so. Yet after a few tries the task is complete and thus there are no crashes until new pictures with previews are added again.

Another way I can trigger the crash is if I open a picture or scroll while the media view is loading. I start the media view, the UI stalls. Then some pictures are shown and the UI is stuttering but responsive for some second. If you tab on a picture in that second, the crash is more likely to happen. If you do not tap on a picture and just wait, the UI stalls again. You can scroll which leads to the android warning that the app does not respond and asks you whether you want to wait, abort or give some feedback. Scrolling during the stall, not stall, stall, not stall process can also cause the crash. Or this was just coincidence - it really is not that clear to me. I just know that if no new pictures are added, media view is a lot more stable and I do not get that crash very often without provoking it.

Sorry that I cannot be more precise.

svenb1234 avatar Jun 23 '22 14:06 svenb1234

The crashes in this issue may be eased by Room, like in the case of #9948, which was fixed when createFileInstance was ported to Room.

Nevertheless this should not close this meta issue: the arbitrary growing fields are fundamentally wrong and must be reworked or they will cause other problems.

AlvaroBrey avatar Jan 07 '23 12:01 AlvaroBrey

Is this also the same issue? This worked on the device before, but stopped after the android upgrade...

Cause of error

Exception in thread "main" java.lang.RuntimeException: Unable to resume activity {com.nextcloud.client/com.owncloud.android.ui.activity.FileDisplayActivity}: android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=17, totalRows=18
    at android.app.ActivityThread.performResumeActivity(ActivityThread.java:5426)
    at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:5507)
    at android.app.servertransaction.ResumeActivityItem.execute(ResumeActivityItem.java:57)
    at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
    at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:180)
    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:98)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2685)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loopOnce(Looper.java:230)
    at android.os.Looper.loop(Looper.java:319)
    at android.app.ActivityThread.main(ActivityThread.java:8913)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:608)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)
Caused by: Exception in thread "main" android.database.sqlite.SQLiteBlobTooBigException: Row too big to fit into CursorWindow requiredPos=17, totalRows=18
    at android.database.sqlite.SQLiteConnection.nativeExecuteForCursorWindow(Native Method)
    at android.database.sqlite.SQLiteConnection.executeForCursorWindow(SQLiteConnection.java:1362)
    at android.database.sqlite.SQLiteSession.executeForCursorWindow(SQLiteSession.java:838)
    at android.database.sqlite.SQLiteQuery.fillWindow(SQLiteQuery.java:62)
    at android.database.sqlite.SQLiteCursor.fillWindow(SQLiteCursor.java:165)
    at android.database.sqlite.SQLiteCursor.onMove(SQLiteCursor.java:126)
    at android.database.AbstractCursor.moveToPosition(AbstractCursor.java:255)
    at android.database.AbstractCursor.moveToNext(AbstractCursor.java:287)
    at com.nextcloud.client.database.dao.FileDao_Impl.getFolderContent(FileDao_Impl.java:1897)
    at com.owncloud.android.datamodel.FileDataStorageManager.getFolderContent(FileDataStorageManager.java:848)
    at com.owncloud.android.datamodel.FileDataStorageManager.getFolderContent(FileDataStorageManager.java:186)
    at com.owncloud.android.ui.adapter.OCFileListAdapter.swapDirectory(OCFileListAdapter.java:690)
    at com.owncloud.android.ui.fragment.OCFileListFragment.listDirectory(OCFileListFragment.java:1331)
    at com.owncloud.android.ui.fragment.OCFileListFragment.listDirectory(OCFileListFragment.java:1296)
    at com.owncloud.android.ui.fragment.OCFileListFragment.listDirectory(OCFileListFragment.java:1285)
    at com.owncloud.android.ui.fragment.OCFileListFragment.onActivityCreated(OCFileListFragment.java:428)
    at androidx.fragment.app.Fragment.performActivityCreated(Fragment.java:3156)
    at androidx.fragment.app.FragmentStateManager.activityCreated(FragmentStateManager.java:619)
    at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:275)
    at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:1943)
    at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:1839)
    at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:1782)
    at androidx.fragment.app.FragmentController.execPendingActions(FragmentController.java:495)
    at androidx.fragment.app.FragmentActivity.onResume(FragmentActivity.java:311)
    at com.owncloud.android.ui.activity.BaseActivity.onResume(BaseActivity.java:91)
    at com.owncloud.android.ui.activity.DrawerActivity.onResume(DrawerActivity.java:1042)
    at com.owncloud.android.ui.activity.FileActivity.onResume(FileActivity.java:253)
    at com.owncloud.android.ui.activity.FileDisplayActivity.onResume(FileDisplayActivity.java:1089)
    at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1603)
    at android.app.Activity.performResume(Activity.java:9119)
    at android.app.ActivityThread.performResumeActivity(ActivityThread.java:5399)
    at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:5507)
    at android.app.servertransaction.ResumeActivityItem.execute(ResumeActivityItem.java:57)
    at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
    at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:180)
    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:98)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2685)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loopOnce(Looper.java:230)
    at android.os.Looper.loop(Looper.java:319)
    at android.app.ActivityThread.main(ActivityThread.java:8913)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:608)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1103)

App information

  • ID: com.nextcloud.client
  • Version: 30270090
  • Build flavor: gplay

Device information

  • Brand: samsung
  • Device: gts9p
  • Model: SM-X816B
  • Id: UP1A.231005.007
  • Product: gts9pxeea

Firmware

  • SDK: 34
  • Release: 14
  • Incremental: X816BXXU1BWK6

MWillimz avatar Jan 21 '24 10:01 MWillimz

This PR can make this problem less likely to happen. Similar discussion.

alperozturk96 avatar Apr 24 '24 09:04 alperozturk96

also with this error here

CssEoCAO avatar Jun 17 '24 10:06 CssEoCAO