commet crash from memory overflow, db corrupted

Describe the bug App ran out of memory and crashed, corrupting the database.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
LineageOS Version: '21.0-20240626-NIGHTLY-enchilada'
Build fingerprint: 'OnePlus/OnePlus6/OnePlus6:8.1.0/OPM1.171019.011/06140300:user/release-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2024-06-30 06:23:30.739634349-0400
Process uptime: 1909s
Cmdline: chat.commet.commetapp
pid: 7696, tid: 3974, name: 79.ui  >>> chat.commet.commetapp <<<
uid: 10194
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
Abort message: '../../third_party/dart/runtime/vm/heap/pages.cc: 1320: error: Out of memory.'
    x0  0000000000000000  x1  0000000000000f86  x2  0000000000000006  x3  0000006bba80dfa0
    x4  752e646c68736d74  x5  752e646c68736d74  x6  752e646c68736d74  x7  7f7f7f7f7f7f7f7f
    x8  00000000000000f0  x9  00000070296b5390  x10 ffffff80fffffbdf  x11 0000000000000000
    x12 0000006bba80b7e0  x13 000000000000004e  x14 0000006bba80b890  x15 0000000000221480
    x16 0000007029760fe0  x17 000000702974c860  x18 0000006b691f8000  x19 0000000000001e10
    x20 0000000000000f86  x21 00000000ffffffff  x22 0000000000000001  x23 0000006e366d8d10
    x24 0000000000000012  x25 0000006d10a70c00  x26 ffffffffffffffff  x27 0000000000000006
    x28 0000000000000000  x29 0000006bba80e020
    lr  00000070296ecad8  sp  0000006bba80df80  pc  00000070296ecb08  pst 0000000000000000
21 total frames
backtrace:
      #00 pc 000000000005ab08  /apex/com.android.runtime/lib64/bionic/libc.so (abort+168) (BuildId: 10f6580e623cb20b4044f6e6a4103b34)
      #01 pc 000000000081432c  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #02 pc 0000000000839728  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #03 pc 0000000000838e7c  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #04 pc 000000000083df1c  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #05 pc 00000000009727d4  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #06 pc 0000000000972f54  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #07 pc 00000000008087b0  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #08 pc 0000000000806d7c  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #09 pc 00000000008068a4  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #10 pc 00000000007a110c  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #11 pc 00000000007b37dc  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #12 pc 0000000000474bf4  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #13 pc 0000000000478784  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #14 pc 000000000000fe04  /system/lib64/libutils.so (android::Looper::pollOnce(int, int*, int*, void**)+680) (BuildId: 16bc30d6f0e04f5d9c5d844339e38a3e)
      #15 pc 000000000001aebc  /system/lib64/libandroid.so (ALooper_pollOnce+100) (BuildId: cd590c13e6fee4592d3652971f80381a)
      #16 pc 0000000000478890  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #17 pc 0000000000476938  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #18 pc 000000000047672c  /data/app/~~I0iObR7ujdnexVLTbhx_Qw==/chat.commet.commetapp-UOvg2ohNHlExr00kFrJUZg==/lib/arm64/libflutter.so (BuildId: e05edcf1368609668cf1873a167462cbcf923136)
      #19 pc 000000000006ac54  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208) (BuildId: 10f6580e623cb20b4044f6e6a4103b34)
      #20 pc 000000000005de68  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 10f6580e623cb20b4044f6e6a4103b34)

Device Information

Device Platform: android Version: v0.3.0-prerelease+5c8379c Git Hash: 5c8379ceaae26187748abb75463edd6135cdb92d Detail: default

System Info

Product: OnePlus6

Jun 30 '24 11:06 theotheroracle

I use commet on Linux Mint and noticed that slowly but surely run out of memory when commet is running. I tracked its total memory consumption with the following command:

while true; do (echo -n "$(date)  "; grep -e '^Pss' /proc/453612/smaps | awk '{sum += $2} END {print sum / 1024, "MB"}'); sleep 300; done

(which basically prints out the total memory consumption every 5 minutes):

Thu 12 Dec 10:06:46 CET 2024  248.524 MB
Thu 12 Dec 10:11:47 CET 2024  264.148 MB
Thu 12 Dec 10:16:47 CET 2024  262.933 MB
Thu 12 Dec 10:21:47 CET 2024  307.812 MB
Thu 12 Dec 10:26:47 CET 2024  323.219 MB
Thu 12 Dec 10:31:47 CET 2024  336.929 MB
Thu 12 Dec 10:36:47 CET 2024  349.882 MB
Thu 12 Dec 10:41:47 CET 2024  332.546 MB
Thu 12 Dec 10:46:47 CET 2024  358.749 MB
Thu 12 Dec 10:51:47 CET 2024  392.819 MB
Thu 12 Dec 10:56:47 CET 2024  405.257 MB
Thu 12 Dec 11:01:47 CET 2024  418.094 MB
Thu 12 Dec 11:06:47 CET 2024  418.729 MB
Thu 12 Dec 11:11:47 CET 2024  415.56 MB
Thu 12 Dec 11:16:47 CET 2024  440.958 MB
Thu 12 Dec 11:21:47 CET 2024  452.289 MB
Thu 12 Dec 11:26:47 CET 2024  453.385 MB
Thu 12 Dec 11:31:47 CET 2024  496.256 MB
Thu 12 Dec 11:36:47 CET 2024  401.993 MB
Thu 12 Dec 11:41:47 CET 2024  354.1 MB
Thu 12 Dec 11:46:48 CET 2024  367.026 MB
Thu 12 Dec 11:51:48 CET 2024  364.567 MB
Thu 12 Dec 11:56:48 CET 2024  391.896 MB
Thu 12 Dec 12:01:48 CET 2024  393.32 MB
Thu 12 Dec 12:06:48 CET 2024  433.318 MB
Thu 12 Dec 12:11:48 CET 2024  448.333 MB
Thu 12 Dec 12:16:48 CET 2024  447.484 MB
Thu 12 Dec 12:21:48 CET 2024  444.207 MB
Thu 12 Dec 12:26:48 CET 2024  460.43 MB
Thu 12 Dec 12:31:48 CET 2024  472.273 MB
Thu 12 Dec 12:36:48 CET 2024  501.343 MB
Thu 12 Dec 12:41:48 CET 2024  511.093 MB
Thu 12 Dec 12:46:48 CET 2024  521.147 MB
Thu 12 Dec 12:51:48 CET 2024  556.385 MB
Thu 12 Dec 12:56:48 CET 2024  549.207 MB
Thu 12 Dec 13:01:48 CET 2024  567.17 MB
Thu 12 Dec 13:06:48 CET 2024  566.523 MB
Thu 12 Dec 13:11:48 CET 2024  591.851 MB
Thu 12 Dec 13:16:48 CET 2024  604.662 MB
Thu 12 Dec 13:21:48 CET 2024  612.177 MB
Thu 12 Dec 13:26:48 CET 2024  618.669 MB
Thu 12 Dec 13:31:48 CET 2024  631.311 MB
Thu 12 Dec 13:36:48 CET 2024  642.998 MB
Thu 12 Dec 13:41:48 CET 2024  656.154 MB
Thu 12 Dec 13:46:48 CET 2024  696.154 MB
Thu 12 Dec 13:51:49 CET 2024  707.958 MB
Thu 12 Dec 13:56:49 CET 2024  706.172 MB
Thu 12 Dec 14:01:49 CET 2024  717.634 MB
Thu 12 Dec 14:06:49 CET 2024  740.323 MB
Thu 12 Dec 14:11:49 CET 2024  758.688 MB
Thu 12 Dec 14:16:49 CET 2024  741.476 MB
Thu 12 Dec 14:21:49 CET 2024  779.12 MB
Thu 12 Dec 14:26:49 CET 2024  791.949 MB
Thu 12 Dec 14:31:49 CET 2024  802.403 MB
Thu 12 Dec 14:36:49 CET 2024  814.901 MB

As you can see, this increased the memory consumption from 250 to 800 MB in 4½ hours. Note this is without interacting with commet at all (it was just sitting there) and without receiving any new messages.

Is there somewhere a memory leak? Any hints how to track that down?

Device Information

Device Platform: linux Version: v0.3.0 Git Hash: 0fc02086a3d641496eb01943ae0c68a97c0b277d Detail: debian

System Info Name: Linux Mint Version: 22 (Wilma)

Dec 12 '24 13:12 sphh

thanks for the info! My first thought is at least part of this is probably coming from the application logs, which are stored in memory indefinitely... I'll change this to be stored to a file, as well as adding an option to disable storing them at all.

I'll have to do some profiling to see if anything else is contributing to this

Dec 15 '24 21:12 Airyzz

How would I be able to see the (in-memory) logs?

Dec 18 '24 09:12 sphh

if you go `Settings > Advanced' and enable Developer Mode, there will be a new tab in settings to access the logs

Dec 18 '24 09:12 Airyzz

@sphh I tried to reproduce this, and running the same command while profiling memory via flutter dev tools, i didn't notice any of the same growth in memory usage.

Any ideas as to what could have been different during your run? are you in any particularly large rooms on your account?

Sat 11 Jan 2025 13:24:13  1138.85 MB
Sat 11 Jan 2025 13:29:13  1169.39 MB
Sat 11 Jan 2025 13:34:13  1141.89 MB
Sat 11 Jan 2025 13:39:13  1158.75 MB
Sat 11 Jan 2025 13:44:13  1087.77 MB
Sat 11 Jan 2025 13:49:13  1095.25 MB
Sat 11 Jan 2025 13:54:13  1109.36 MB
Sat 11 Jan 2025 13:59:13  1127.88 MB
Sat 11 Jan 2025 14:04:13  1144.68 MB
Sat 11 Jan 2025 14:09:13  1141.97 MB
Sat 11 Jan 2025 14:14:14  1158.59 MB
Sat 11 Jan 2025 14:19:14  1149.79 MB
Sat 11 Jan 2025 14:24:14  1159.29 MB
Sat 11 Jan 2025 14:29:14  1155.84 MB
Sat 11 Jan 2025 14:34:14  1146.46 MB
Sat 11 Jan 2025 14:39:14  1160.38 MB
Sat 11 Jan 2025 14:44:14  1159.05 MB
Sat 11 Jan 2025 14:49:14  1165.48 MB
Sat 11 Jan 2025 14:54:14  1162.54 MB
Sat 11 Jan 2025 14:59:14  1147.66 MB
Sat 11 Jan 2025 15:04:14  1161.45 MB
Sat 11 Jan 2025 15:09:14  1154.7 MB
Sat 11 Jan 2025 15:14:14  1170.43 MB
Sat 11 Jan 2025 15:19:14  1184.29 MB
Sat 11 Jan 2025 15:24:14  1171.23 MB
Sat 11 Jan 2025 15:29:14  1181.86 MB
Sat 11 Jan 2025 15:34:14  1155.96 MB
Sat 11 Jan 2025 15:39:14  1151.84 MB
Sat 11 Jan 2025 15:44:14  1181.91 MB
Sat 11 Jan 2025 15:49:14  1152.65 MB
Sat 11 Jan 2025 15:54:14  1117.76 MB
Sat 11 Jan 2025 15:59:14  1129.94 MB
Sat 11 Jan 2025 16:04:14  1121.51 MB
Sat 11 Jan 2025 16:09:14  1117.31 MB
Sat 11 Jan 2025 16:14:14  1125.45 MB
Sat 11 Jan 2025 16:19:14  1112.12 MB
Sat 11 Jan 2025 16:24:14  1125.58 MB
Sat 11 Jan 2025 16:29:14  1114.83 MB
Sat 11 Jan 2025 16:34:14  1124.46 MB
Sat 11 Jan 2025 16:39:14  1117.73 MB
Sat 11 Jan 2025 16:44:14  1127.96 MB
Sat 11 Jan 2025 16:49:14  1121.23 MB

Jan 11 '25 07:01 Airyzz

I have

four accounts
6 private chats
3 rooms: one (still) empty, the others: #öffi:schildbach.de and #tammy:imbitbu.de

I installed commet from the .deb file, if that makes any change.

Jan 11 '25 11:01 sphh

@Airyzz: Is there any progress in this respect? Can I help with more information?

Jan 31 '25 08:01 sphh

Is there any progress in this respect? Can I help with more information?

I need to be able to replicate the behaviour, so if you are able to figure out if this is only occurring while logged in to just one of your multiple accounts, or if its activity in a room, that will help to narrow it down

Feb 01 '25 00:02 Airyzz

I tried to isolate the problem. I removed all accounts but the matrix.org account. The memory consumption increased up to a certain value and then after roughly 3 minutes jumped back to a lower value. After that the memory consumption always oscillated between a lower and higher value. These values did not change much, and what is most important, the upper limit did not increase.

I then added back the other accounts one by one. Now I don't see that continuously increasing memory consumption any more.

I am puzzled.

Feb 03 '25 18:02 sphh

I just experienced another continuously increasing memory consumption:

I am subscribed to the public room #öffi:schildbach.de
There opened the thread starting on 4.2.2025 at 22:42 (CET). The message is from me („stephan“) and starts with “Ich habe in paar Probleme mit den „Nearby Stations“:”
The I move the mouse over the last message (6.2.2025, 00:35 CET). The popup menu shows (no idea if that is important) and I left commet running.
Memory consumption increases about 180MB in 3 hours time.

Feb 06 '25 14:02 sphh

Interesting, how would you go building commet from source? if you did, you would be able to make use of flutter's memory profiler, which would give the most insight - happy to help walk you through it if you need.

Feb 07 '25 11:02 Airyzz

I did not compile commet. I use the debian build. Could YOU provide a version with the memory profiler enabled?

Feb 07 '25 12:02 sphh

I don't believe I can, its only accessible from the flutter dev tools. to do it you would have to attempt to build it yourself, but it would provide probably the most useful information

Feb 07 '25 12:02 Airyzz

I searched for flutter in my package manager. Sadly nothing turned up …

Have you tried that room and thread yourself? Do you also see the memory used increasing?

Feb 07 '25 12:02 sphh

I haven't had a chance to look at it just yet, but will try to do so soon.

if you do want to give it a try, here is the docs for installing flutter, but i will try to investigate it myself regardless

Feb 07 '25 13:02 Airyzz

Same for the latest version v0.3.1-prerelease+952499f 😢.

But I noticed something with Tammy: This Matrix client shows an error message, that it could not connect to some of my accounts. The developer believes, that ‘it seems like the homeserver is violating the spec (sends 403 instead of 401)’ and suspects synapse. He fixed it here for Tammy.

Could this be the reason for the increasing memory consumption?

The accounts affected were on converser.eu, but also on matrix.org.

The error message in Tammy was:

08:52:15.727 [DefaultDispatcher-worker-15] ERROR n.f.t.c.client.SyncApiClient - error while sync with token […]
net.folivo.trixnity.core.MatrixServerException: statusCode: 403 ; errorResponse: Forbidden(error=refresh token isn't valid anymore)
        at net.folivo.trixnity.clientserverapi.client.SyncApiClientImpl.sync-InPyf_0(SyncApiClient.kt:475)
        at net.folivo.trixnity.clientserverapi.client.SyncApiClientImpl<a hr<a href="https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$1.invokeSuspend">https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$1.invokeSuspend</a>ix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$sync">https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$sync</a>$1.invokeSuspend(SyncApiClient.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:35)
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:101)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeW<a href="https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$Worker.executeTask">https://matrix.to/#/!dAdM<a href="https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$Worker.runWorker">https://matrix.to/#/!dAdM<a href="https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$Worker.run">https://matrix.to/#/!dAdMsQExaTsVzKcYhH:imbitbu.de/$Worker.run</a>YhH:imbitbu.de/$Worker.runWorker</a>bitbu.de/$Worker.executeTask</a>l.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:98)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

Feb 19 '25 10:02 sphh

commet commet copied to clipboard

crash from memory overflow, db corrupted

commet
commet copied to clipboard