[Memory Error]: Scudo OOM when running LLM on S24 Ultra or any latest android launched with android 14 or above

[Nick-infinity]

Hello, I am trying to run LLM on s24 Ultra device with 12GB of ram. My LLM has a large embedding size of 160984 * 2048 . The fp32 file size is 1.2GB . When MNN-LLM tries to load it via Express:Module:load api, https://github.com/alibaba/MNN/blob/ffde3f423da2f0ccd715e86b8c8e9f4e5c359060/source/core/FileLoader.cpp#L55 this line throws error and the memory allocation fails. I checked the malloc and it fails due to OOM. I believe there are excessive mmap calls in the MNN system. I have attached pastebin link for the same.

Interesting thing is that same code binaries and model works on s23 ultra device. I am guessing memory management spec got changed with device launching with android 14 or greater but not very sure.

Can someone help me with this. I am sure the qwen1.5-1.8B model will fail on s24 ultra / pixel 8 device if fp32 embedding is used. Fp16 embedding works fine though but speeds of fp32 embedding is much faster than fp16 embedding in my case. My model is 2GB after 4bit MNN Quantization

@wangzhaode @v0jiuqi

平台(如果交叉编译请再附上交叉编译目标平台):

Platform(Include target platform as well if cross-compiling): Host : MacOs , Target : Android

Github版本:

Github Version: 2.8.3 MNN

直接下载ZIP包请提供下载日期以及压缩包注释里的git版本(可通过7z l zip包路径命令并在输出信息中搜索Comment 获得，形如Comment = bc80b11110cd440aacdabbf59658d630527a7f2b)。 git clone请提供 git commit 第一行的commit id

Provide date (or better yet, git revision from the comment section of the zip. Obtainable using 7z l PATH/TO/ZIP and search for Comment in the output) if downloading source as zip,otherwise provide the first commit id from the output of git commit

编译方式:

Compiling Method

请在这里粘贴cmake参数或使用的cmake脚本路径以及完整输出
Paste cmake arguments or path of the build script used here as well as the full log of the cmake proess here or pastebin
MNN/project/android/../build_64.sh -DMNN_LOW_MEMORY=ON -DMNN_ARM82=ON
NDK: r21
Sample Build Script: https://github.com/wangzhaode/mnn-llm/blob/master/script/android_app_build.sh
Same issue is there with MNN 2.8.1 or greater. The issue is more on the Android 14 changes as the same code works on other devices

编译日志:

Build Log:

粘贴在这里
Paste log here or pastebin : https://pastebin.com/iN6rUfcc

[github-actions[bot]]

Marking as stale. No activity in 60 days.