sdmaid-se icon indicating copy to clipboard operation
sdmaid-se copied to clipboard

Published 20 hours ago • verified publisher icon d4rken-org

Storage analysis hangs at scanning WhatsApp

Open Girofox opened this issue 1 year ago • 5 comments

Describe the bug When starting storage analyser, the scan hangs at WhatsApp. Even waiting 20 minutes doesn't help. My WhatsApp folder is very large with 13 GB but is on internal storage which should be fast. I already did excluded WhatsApp under app exclusions (path exclusion didn't worked because this triggers a scan).

Im SD Maid 1 this isn't a problem, the scan just takes a minute.

To Reproduce Steps to reproduce the behavior:

  1. Go to 'Storage analysis
  2. Click on internal storage / main storage
  3. Wait for scan to start
  4. Scan hangs at WhatsApp

Expected behavior Scan should resume after some time. Maybe there should be an option to exclude individual folders for storage analysis.

Smartphone (please complete the following information):

  • Device: Samsung Galaxy S10
  • OS: Android 12 (One UI 4.1)
  • Non Root
  • SD Maid 2/SE version e.g. v0.19.0-beta0

Screenshots Screenshot_20240128_205435_SD Maid SE

Additional context WhatsApp folder at 13 GB which contains many video files. Can't add the path exclusion because this triggers a scan which causes the hang.

Girofox avatar Jan 28 '24 20:01 Girofox

Can you provide a debug log of this?

I'm preparing some performance improvements here but i'm not sure if that covers your case.

d4rken avatar Jan 28 '24 22:01 d4rken

Can you provide a debug log of this?

I'm preparing some performance improvements here but i'm not sure if that covers your case.

This is the part of the log file where it gets stuck (note the size of WhatsApp Images folder:

2024-01-29T12:36:01.683Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Profile Photos), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-16T22:25:22Z, target=null)
2024-01-29T12:36:01.683Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Images), fileType=DIRECTORY, size=532480, modifiedAt=2024-01-29T08:56:16Z, target=null)
2024-01-29T12:36:05.361Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Images)

Girofox avatar Jan 29 '24 13:01 Girofox

And here is the beginning section of WhatsApp scan. There are some error messages but don't know if they are relevant. Maybe i can provide the same debug log for SD Maid 1 as a comparison.

2024-01-29T12:36:01.158Z  V/SDMSE:SAF:Mapper: Trying to match volume StorageVolumeX(uuid=null, directory=/storage/emulated/0, userlabel=Interner Speicher, volumeX=StorageVolume: Interner Speicher, rootUri=content://com.android.externalstorage.documents/root/primary) against LocalPath(/storage/emulated/0/Android/data/com.whatsapp)
2024-01-29T12:36:01.159Z  V/SDMSE:SAF:Mapper: Trying to match volume StorageVolumeX(uuid=7F0A-229A, directory=/storage/7F0A-229A, userlabel=SD-Karte, volumeX=StorageVolume: SD-Karte (7F0A-229A), rootUri=content://com.android.externalstorage.documents/root/7F0A-229A) against LocalPath(/storage/emulated/0/Android/data/com.whatsapp)
2024-01-29T12:36:01.159Z  V/SDMSE:SAF:Mapper: Target storageVolumes for LocalPath(/storage/emulated/0/Android/data/com.whatsapp) is StorageVolumeX(uuid=null, directory=/storage/emulated/0, userlabel=Interner Speicher, volumeX=StorageVolume: Interner Speicher, rootUri=content://com.android.externalstorage.documents/root/primary)
2024-01-29T12:36:01.159Z  V/SDMSE:SAF:Mapper: toSAFPath() LocalPath(/storage/emulated/0/Android/data/com.whatsapp) -> SAFPath(treeRoot=content://com.android.externalstorage.documents/tree/primary, segments=[Android, data, com.whatsapp])
2024-01-29T12:36:01.160Z  V/SDMSE:Gateway:SAF: No UriPermission match for SAFPath(treeRoot=content://com.android.externalstorage.documents/tree/primary, segments=[Android, data, com.whatsapp])
2024-01-29T12:36:01.160Z  W/SDMSE:Gateway:SAF: lookup(SAFPath(treeRoot=content://com.android.externalstorage.documents/tree/primary, segments=[Android, data, com.whatsapp])) failed.
2024-01-29T12:36:01.163Z  V/SDMSE:Gateway:Local: exists(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.164Z  V/SDMSE:Analyzer:Storage:Scanner:Extensions: Walking content items for LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.166Z  V/SDMSE:Gateway:Local: lookup(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.169Z  V/SDMSE:Gateway:Local: Looked up: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-16T22:24:51Z, target=null)
2024-01-29T12:36:01.172Z  V/SDMSE:Gateway:Local: lookup(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.176Z  V/SDMSE:Gateway:Local: Looked up: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-16T22:24:51Z, target=null)
2024-01-29T12:36:01.181Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.188Z  V/SDMSE:Gateway:Local: Looked up 1 items:
2024-01-29T12:36:01.189Z  V/SDMSE:Gateway:Local: #0 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-19T01:51:21Z, target=null)
2024-01-29T12:36:01.189Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-19T01:51:21Z, target=null)
2024-01-29T12:36:01.198Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp)
2024-01-29T12:36:01.237Z  V/SDMSE:Gateway:Local: Looked up 7 items:
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #0 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Backups), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:28Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #1 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.trash), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T08:56:13Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #2 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.StickerThumbs), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-22T20:11:12Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #3 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Databases), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:25Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #4 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media), fileType=DIRECTORY, size=4096, modifiedAt=2023-05-25T12:50:29Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Local: #5 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Thumbs), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-17T07:07:35Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Local: #6 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Shared), fileType=DIRECTORY, size=12288, modifiedAt=2024-01-29T12:25:28Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Backups), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:28Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.trash), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T08:56:13Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.StickerThumbs), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-22T20:11:12Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Databases), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:25Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media), fileType=DIRECTORY, size=4096, modifiedAt=2023-05-25T12:50:29Z, target=null)
2024-01-29T12:36:01.240Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Thumbs), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-17T07:07:35Z, target=null)
2024-01-29T12:36:01.240Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Shared), fileType=DIRECTORY, size=12288, modifiedAt=2024-01-29T12:25:28Z, target=null)
2024-01-29T12:36:01.255Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Shared)

Girofox avatar Jan 29 '24 13:01 Girofox

Update:

i updated to the latest version v0.19.1-beta0 and the scan finishes after 5 minutes. At least it seems to work now.

Still slower compared to SD maid 1 though. The issue seems that my WhatsApp Images folder containd 20k elements inside (not nested though), whose sizes are on average below 500 kB.

Don't know if it is possible to multithread a scan of folder with large number of files. In SD Maid 1 the multithread setting didn't affected media scan.

Girofox avatar Jan 29 '24 13:01 Girofox

Hm 0.19.0 -> 0.19.1 had no changes with regards to SD Maid's IO performance 🤷

Can you record a debuglog for a little bit, with trace mode enabled? I'm wondering where the bottleneck is, the initial File.listFiles() in normal access mode is probably as fast as it gets, if the additional look up of file attributes per File returned by File.listFiles() is the bottleneck, then this could potentially be parallelized, whether that helps depends on the storage hardware 🤔 , might be worth a try tho..

WhatsApp folder at 13 GB which contains many video files. Can't add the path exclusion because this triggers a scan which causes the hang.

Exclusions don't affect the analyzer as it would falsify the size calculations. I have some ideas about changing this top be more "scan subfolders on demand". 🧑‍🏭

Total folder size does not affect performance, but the amount of files does, 20k is quite a lot.

d4rken avatar Feb 04 '24 11:02 d4rken

Closing this as it seems to be fixed. Some performance updates also happened in the meantime.

d4rken avatar May 02 '24 13:05 d4rken