llama.cpp How to run in termux

I solved this problem through ai.

/data/data/com.termux/files/home/llamaqnn/ggml/src/ggml-qnn/utils.cpp:253:17: error: reference to unresolved using declaration 253 | return std::aligned_alloc(alignment, size); | ^ /data/data/com.termux/files/usr/include/c++/v1/cstdlib:150:1: note: using declaration annotated with 'using_if_exists' here 150 | using ::aligned_alloc _LIBCPP_USING_IF_EXISTS; | ^ /data/data/com.termux/files/home/llamaqnn/ggml/src/ggml-qnn/utils.cpp:253:12: error: excess elements in scalar initializer 253 | return std::aligned_alloc(alignment, size); | ^ ~~~~~~ 2 errors generated. make[2]: *** [ggml/src/ggml-qnn/CMakeFiles/ggml-qnn.dir/build.make:177: ggml/src/ggml-qnn/CMakeFiles/ggml-qnn.dir/utils.cpp.o] Error 1 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [CMakeFiles/Makefile2:2370: ggml/src/ggml-qnn/CMakeFiles/ggml-qnn.dir/all] Error 2

Then how to let backend find correct lib. I am using Xiaomi 14 Pro

Mar 02 '25 05:03 belog2867

he full capacity of the model will not be utilized extend_lib_search_path is nullptr, will use . as defaultunable to find a valid qnn system interface failed to load QNN system lib failed to init qnn backend qnn-gpu解

Mar 02 '25 05:03 belog2867

oh, do you mind build it with offical ndk-toolchain on your host machine? thats the official way google suggested, and its clang is relatively new and support the newer c/c++ standard

Can have a look here, hope it helps: https://github.com/chraac/llama.cpp/wiki/How-to-Build

Mar 03 '25 13:03 chraac

I set the location of the library, but it doesn't seem to be used. Can I locate these libraries in the approximate location of /vendor/lib64?

extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so can not load QNN library libQnnSystem.so, error: dlopen failed: library "libQnnSystem.so" not found failed to load QNN system lib failed to init qnn backend qnn-gpu llama_init_from_model: failed to initialize qnn-gpu backend

This is my build command. cd llama.cpp &&
mkdir build-android && cd build-android

cmake .. -G Ninja
-DCMAKE_TOOLCHAIN_FILE=$HOME/android-sdk/ndk/26.3.11579264/build/cmake/android.toolchain.cmake
-DANDROID_ABI=arm64-v8a
-DANDROID_PLATFORM=android-28
-DBUILD_SHARED_LIBS=OFF
-DGGML_RPC=ON
-DGGML_QNN=ON
-DGGML_QNN_SDK_PATH=~/qairt/2.31.0.250130
-DGGML_OPENMP:BOOL=OFF
-DGGML_QNN_DEFAULT_LIB_SEARCH_PATH=/vendor/lib64

ninja

Mar 04 '25 15:03 belog2867

At the best of times, it told me that libQnnGpu.so was missing.

Mar 04 '25 15:03 belog2867

actually, you have to push the qnn related dynamic library to you cellphone before run, a complete list of those libs can be found here: https://github.com/chraac/llama-cpp-qnn-builder/blob/dd7ba303a8e3213c8cafe330c0938b25c6bd788f/docker/build_in_container.sh#L83

Mar 04 '25 15:03 chraac

How to make this backend use lib in the./ directory?

Mar 05 '25 13:03 belog2867

maybe you can try to overwrite the LD_LIBRARY_PATH env var

Mar 05 '25 14:03 chraac

Now it gives me a new problem.extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so can not load QNN library libQnnSystem.so, error: dlopen failed: library "libstdc++.so.6" not found: needed by /data/data/com.termux/files/home/a/libQnnSystem.so in namespace (default)

Missing cpp library? ? Thank you for your help.

Mar 05 '25 14:03 belog2867

hmmm, may I ask which Android version and device model you're using? looks like libQnnSystem.so relies on libstdc++.so.6 and has failed to load

Mar 06 '25 02:03 chraac

android 15 xiaomi 14 pro hyperos2.0.104.0

Mar 06 '25 11:03 belog2867

android 15 xiaomi 14 pro hyperos2.0.104.0

interesting.... my phone is also android 15, but there's no such error come out....

and this is symbol table dump of my libQnnSystem.so

libQnnSystem.so:     file format elf64-little

DYNAMIC SYMBOL TABLE:
0000000000000000      DF *UND*  0000000000000000 (LIBC)       __cxa_finalize
0000000000000000      DF *UND*  0000000000000000 (LIBC)       __cxa_atexit
0000000000000000      DF *UND*  0000000000000000 (LIBC)       strncmp
0000000000000000      DF *UND*  0000000000000000 (LIBC)       posix_memalign
0000000000000000      DF *UND*  0000000000000000 (LIBC)       syslog
0000000000000000      DF *UND*  0000000000000000 (LIBC)       snprintf
0000000000000000      DF *UND*  0000000000000000 (LIBC)       openlog
0000000000000000      DF *UND*  0000000000000000 (LIBC)       free
0000000000000000      DF *UND*  0000000000000000 (LIBC)       memchr
0000000000000000      DF *UND*  0000000000000000 (LIBC)       fwrite
0000000000000000      DF *UND*  0000000000000000 (LIBC)       realloc
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_getspecific
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_rwlock_unlock
0000000000000000      DF *UND*  0000000000000000 (LIBC)       memcpy
0000000000000000      DF *UND*  0000000000000000 (LIBC)       strndup
0000000000000000      DF *UND*  0000000000000000 (LIBC)       __errno
0000000000000000      DF *UND*  0000000000000000 (LIBC)       memmove
0000000000000000      DF *UND*  0000000000000000 (LIBC)       __system_property_get
0000000000000000      DF *UND*  0000000000000000 (LIBC)       memset
0000000000000000      DF *UND*  0000000000000000 (LIBC)       vfprintf
0000000000000000      DF *UND*  0000000000000000 (LIBC)       malloc
0000000000000000      DF *UND*  0000000000000000 (LIBC)       fprintf
0000000000000000      DF *UND*  0000000000000000 (LIBC)       abort
0000000000000000      DF *UND*  0000000000000000 (LIBC)       closelog
0000000000000000      DF *UND*  0000000000000000 (LIBC)       strlen
0000000000000000      DF *UND*  0000000000000000 (LIBC)       getauxval
0000000000000000      DO *UND*  0000000000000000 (LIBC)       __sF
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_cond_broadcast
0000000000000000      DF *UND*  0000000000000000 (LIBC)       strcmp
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_rwlock_rdlock
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_key_delete
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_setspecific
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_cond_wait
0000000000000000      DF *UND*  0000000000000000 (LIBC)       strerror_r
0000000000000000      DF *UND*  0000000000000000 (LIBC)       syscall
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_once
0000000000000000      DF *UND*  0000000000000000 (LIBC)       vasprintf
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_mutex_lock
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_rwlock_wrlock
0000000000000000      DF *UND*  0000000000000000 (LIBC)       android_set_abort_message
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_mutex_destroy
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_mutex_unlock
0000000000000000      DF *UND*  0000000000000000 (LIBC)       getpid
0000000000000000      DF *UND*  0000000000000000 (LIBC)       pthread_key_create
0000000000000000      DF *UND*  0000000000000000 (LIBC)       __assert2
0000000000000000      DF *UND*  0000000000000000 (LIBC)       fflush
0000000000000000      DF *UND*  0000000000000000 (LIBC)       fputc
0000000000000000      DF *UND*  0000000000000000 (LIBC)       dl_iterate_phdr
0000000000019f10 g    DF .text  000000000000002c  Base        QnnSystemInterface_getProviders

as you can see, there's no relied deps with libstdc++ wondering where did you get your libQnnSystem.so? it should be from <qnn_sdk_path>/qairt/2.31.0.250130/lib/aarch64-android for android

Mar 07 '25 10:03 chraac

Oh I got them in QNN_SDK_PATH/lib/aarch64-oe-linux-gcc11.2 and QNN_SDK_PATH/lib/hexagon-v*

Mar 07 '25 11:03 belog2867

I got another problem,maybe I'm not far from success.

he full capacity of the model will not be utilized extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so initialize qnn system successfully failed to load /data/local/tmp/libQnnGpu.so, fallback to libQnnGpu.so can not open QNN library libQnnGpu.so, with error: dlopen failed: cannot locate symbol "Xzs_Construct" referenced by "/system/lib64/libunwindstack.so"... failed to load QNN backend failed to init qnn backend qnn-gpu llama_init_from_model: failed to initialize qnn-gpu backend common_init_from_params: failed to create context with model '../.ollama/models/blobs/sha256-d547a29843982f182448eeacc5d3a9ddfb0f29f88d986b7339f5b533477023ac' main: error: unable to load model

I have this lib in /system/lib64.

Mar 07 '25 11:03 belog2867

I tried redmi k80pro snapdragon 8elite it has the same problem.

Mar 07 '25 13:03 belog2867

please make sure copy those so from the correct folder first as i said before, can have a look on my compile script: https://github.com/chraac/llama-cpp-qnn-builder/blob/dd7ba303a8e3213c8cafe330c0938b25c6bd788f/docker/build_in_container.sh#L35

Mar 07 '25 13:03 chraac

Before, you gave me a different path. <qnn_sdk_path>/qairt/2.31.0.250130/lib/aarch64-android There will be a problem with this path: about Xzs_Construct

The path in the build script can't even load libqnnsystem.so

Mar 07 '25 14:03 belog2867

Use aarch64-android director can load libqnnsystem.so then gave me error can not open QNN library libQnnGpu.so, with error: dlopen failed: cannot locate symbol "Xzs_Construct" referenced by "/system/lib64/libunwindstack.so"

Mar 07 '25 14:03 belog2867

hmmm, i mean the required shared object (.so) files should be copied from the QNN SDK path at: <qnn_sdk_path>/qairt/2.31.0.250130/lib/aarch64-android

where <qnn_sdk_path> refers to the base directory of your QNN SDK root folder. this location contains all the necessary runtime libraries needed for the Android arm64 platform.

Mar 07 '25 14:03 chraac

Also, can you speak Chinese? I have some difficulties in using English. I heard that you are Chinese in other places, and so am I.

Mar 07 '25 14:03 belog2867

yeah, sure, feel free to use chinese here

Mar 07 '25 14:03 chraac

我使用了这个文件夹中的库，aarch64-oe-linux-gcc11.2这个文件夹的库连libqnnsystem都无法加载，而图中的可以加载，到libqnngpu报错了

Mar 07 '25 14:03 belog2867

问下你怎么跑的程序？看到你这里的几个lib和我本地的版本大小都不一样，应该是复制错了

Mar 07 '25 14:03 chraac

这个呢

Mar 07 '25 14:03 belog2867

然后再复制这个目录里的正则表达式libqnnhtp*的一些文件

Mar 07 '25 14:03 belog2867

然后还是这个问题，好像这个库缺点啥

Mar 07 '25 14:03 belog2867

运行命令LD_LIBRARY_PATH=/vendor/lib64:$PREFIX/lib:./ ./llama-cli -m ../.ollama/models/blobs/sha256-d547a29843982f182448eeacc5d3a9ddfb0f29f88d986b7339f5b533477023ac -cnv

Mar 07 '25 14:03 belog2867

新报错 full capacity of the model will not be utilized extend_lib_search_path is nullptr, will use . as defaultdevice property is not supported create QNN device successfully failed to load ./libcdsprpc.so, fallback to libcdsprpc.so qnn device name qnn-gpu extend_lib_search_path is nullptr, will use . as defaultdevice counts 1 deviceID:0, deviceType:0, numCores 1 htp_type:0(ON_CHIP) qualcomm soc_model:57(SD 8 Gen 3 (SM8650)), htp_arch:75(QCOM_HTP_V75), vtcm_size:8 MB failed to create QNN device failed to load ./libcdsprpc.so, fallback to libcdsprpc.so why failed to initialize qnn context failed to init qnn backend qnn-npu llama_init_from_model: failed to initialize qnn-npu backend idx 1, name:qnn-gpu

Mar 07 '25 15:03 belog2867

好像我的qnn ndk太老了，更新完，现在出新问题了( llama_init_from_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so initialize qnn system successfully failed to load /data/local/tmp/libQnnGpu.so, fallback to libQnnGpu.so device property is not supported create QNN device successfully failed to load /data/local/tmp/libcdsprpc.so, fallback to libcdsprpc.soqnn device name qnn-gpu extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so initialize qnn system successfully failed to load /data/local/tmp/libQnnHtp.so, fallback to libQnnHtp.so device counts 1 deviceID:0, deviceType:0, numCores 1 htp_type:0(ON_CHIP) qualcomm soc_model:57(SD 8 Gen 3 (SM8650)), htp_arch:75(QCOM_HTP_V75), vtcm_size:8 MB create QNN device successfully failed to load /data/local/tmp/libcdsprpc.so, fallback to libcdsprpc.sofailed to allocate rpc memory, size: 2048 MB capacity of QNN rpc ion memory is about 2000 MB HTP backend perf_infrastructure creation ok HTP infra type = 0, which is perf infra type qnn device name qnn-npu llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1 llama_kv_cache_init: qnn-gpu KV buffer size = 112.00 MiB llama_init_from_model: KV self size = 112.00 MiB, K (f16): 56.00 MiB, V (f16): 56.00 MiB llama_init_from_model: CPU output buffer size = 0.58 MiB llama_init_from_model: qnn-gpu compute buffer size = 317.25 MiB llama_init_from_model: CPU compute buffer size = 17.51 MiB llama_init_from_model: graph nodes = 986 llama_init_from_model: graph splits = 452 common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) /data/data/com.termux/files/home/llamaqnn/ggml/src/ggml-qnn/op-config-caps.cpp:382: GGML_ASSERT(kOpCaps[op_index].qnn_op_name) failed

分配rpc内存失败

Mar 08 '25 11:03 belog2867

我的启动命令 LD_LIBRARY_PATH=/vendor/lib64:/system/lib64 ./llama-cli -m ~/.ollama/models/blobs/sha256-d547a29843982f182448eeacc5d3a9ddfb0f29f88d986b7339f5b533477023ac -cnv -ngl 99

Mar 08 '25 11:03 belog2867

好像我的qnn ndk太老了，更新完，现在出新问题了( llama_init_from_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so initialize qnn system successfully failed to load /data/local/tmp/libQnnGpu.so, fallback to libQnnGpu.so device property is not supported create QNN device successfully failed to load /data/local/tmp/libcdsprpc.so, fallback to libcdsprpc.soqnn device name qnn-gpu extend_lib_search_path is nullptr, will use /data/local/tmp as defaultfailed to load /data/local/tmp/libQnnSystem.so, fallback to libQnnSystem.so initialize qnn system successfully failed to load /data/local/tmp/libQnnHtp.so, fallback to libQnnHtp.so device counts 1 deviceID:0, deviceType:0, numCores 1 htp_type:0(ON_CHIP) qualcomm soc_model:57(SD 8 Gen 3 (SM8650)), htp_arch:75(QCOM_HTP_V75), vtcm_size:8 MB create QNN device successfully failed to load /data/local/tmp/libcdsprpc.so, fallback to libcdsprpc.sofailed to allocate rpc memory, size: 2048 MB capacity of QNN rpc ion memory is about 2000 MB HTP backend perf_infrastructure creation ok HTP infra type = 0, which is perf infra type qnn device name qnn-npu llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1 llama_kv_cache_init: qnn-gpu KV buffer size = 112.00 MiB llama_init_from_model: KV self size = 112.00 MiB, K (f16): 56.00 MiB, V (f16): 56.00 MiB llama_init_from_model: CPU output buffer size = 0.58 MiB llama_init_from_model: qnn-gpu compute buffer size = 317.25 MiB llama_init_from_model: CPU compute buffer size = 17.51 MiB llama_init_from_model: graph nodes = 986 llama_init_from_model: graph splits = 452 common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) /data/data/com.termux/files/home/llamaqnn/ggml/src/ggml-qnn/op-config-caps.cpp:382: GGML_ASSERT(kOpCaps[op_index].qnn_op_name) failed

分配rpc内存失败

能问下用的啥模型吗？看起来是assert了

Mar 08 '25 12:03 chraac