[Bug] ValueError: Check failed: (f != nullptr) is false: Cannot find function vm.builtin.kv_state_clear

[NSTiwari]

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

FINE_TUNED_MODEL_NAME = "fine_tuned_science_gemma2b-it"

QUANTIZATION= "q4f16_1"

!python -m mlc_chat convert_weight /content/$FINE_TUNED_MODEL_NAME/ --quantization $QUANTIZATION -o /content/$FINE_TUNED_MODEL_NAME-$QUANTIZATION-MLC/

!mlc_chat gen_config /content/$FINE_TUNED_MODEL_NAME/ --quantization $QUANTIZATION \ --conv-template gemma_instruction --context-window-size 768 -o /content/$FINE_TUNED_MODEL_NAME-$QUANTIZATION-MLC/

!mlc_chat compile /content/$FINE_TUNED_MODEL_NAME-$QUANTIZATION-MLC/mlc-chat-config.json \ --device android -o /content/$FINE_TUNED_MODEL_NAME-$QUANTIZATION-MLC/$FINE_TUNED_MODEL_NAME-$QUANTIZATION-android.tar

Executed the prepare_libs.sh file.

Built the Android app by placing the JAR files in library.

Error:

app-config.json mlc-chat-config.json

Expected behavior

Android app fails to run. It crashes with the error ValueError: Check failed: (f != nullptr) is false: Cannot find function vm.builtin.kv_state_clear.

Environment

• Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA):
• Operating system (e.g. Ubuntu/Windows/MacOS/...):
• Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...)
• How you installed MLC-LLM (conda, source):
• How you installed TVM-Unity (pip, source):
• Python version (e.g. 3.10):
• GPU driver version (if applicable):
• CUDA/cuDNN version (if applicable):
• TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):
• Any other relevant information:

Additional context

[MasterJH5574]

Hey @NSTiwari thanks for reporting. It looks to me that this is because your TVM is not the latest. To help confirm, could you check if the line 34 of file 3rdparty/tvm/src/runtime/relax_vm/kv_state.cc is this line? https://github.com/apache/tvm/blob/f8b9a5faa440d594d125271383ebef4037a106ac/src/runtime/relax_vm/kv_state.cc#L34

If it is not, or this file does not exist, then it means your TVM is not the latest. You will need to do submodule update by

git submodule update --init --recursive

and run prepare_libs.sh and subsequent steps again.

[NSTiwari]

@MasterJH5574: Thanks for the suggestion. Unfortunately, my computer doesn't support CMake and I've tried a lot with other devices as well but there are some compatibility issues with Java. Could you please run the prepare_libs.sh script on the latest TVM for the below app-config.json and the Android .tar file and provide me the output JAR files? I would really appreciate your help.

app-config.json prebuilt.zip

[RageshAntonyHM]

@NSTiwari @MasterJH5574

after updating all modules, when running ./prepare_libs.sh , I am getting this error:

Traceback (most recent call last):
  File "/Users/rageshantonyd/Documents/working_prj/mlc-llm/android/library/prepare_model_lib.py", line 78, in <module>
    main()
  File "/Users/rageshantonyd/Documents/working_prj/mlc-llm/android/library/prepare_model_lib.py", line 37, in main
    available_model_libs = get_model_libs(lib_path)
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rageshantonyd/Documents/working_prj/mlc-llm/android/library/prepare_model_lib.py", line 7, in get_model_libs
    global_symbol_map = ndk.get_global_symbol_section_map(lib_path)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'tvm.contrib.ndk' has no attribute 'get_global_symbol_section_map'

I am using Mac M2

I updated TVM to latest commit #16812

pip show tvm output states

Name: tvm Version: 0.16.dev984+g2dcf9ec5a

[NSTiwari]

@NSTiwari @MasterJH5574

after updating all modules, when running ./prepare_libs.sh , I am getting this error:

Traceback (most recent call last):
  File "/Users/rageshantonyd/Documents/working_prj/mlc-llm/android/library/prepare_model_lib.py", line 78, in <module>
    main()
  File "/Users/rageshantonyd/Documents/working_prj/mlc-llm/android/library/prepare_model_lib.py", line 37, in main
    available_model_libs = get_model_libs(lib_path)
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rageshantonyd/Documents/working_prj/mlc-llm/android/library/prepare_model_lib.py", line 7, in get_model_libs
    global_symbol_map = ndk.get_global_symbol_section_map(lib_path)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'tvm.contrib.ndk' has no attribute 'get_global_symbol_section_map'

I am using Mac M2

I updated TVM to latest commit #16812

pip show tvm output states

Name: tvm Version: 0.16.dev984+g2dcf9ec5a

@MasterJH5574: Could you have a look?

[NSTiwari]

@RageshAntonyHM: Probably, there's an update in the MLC LLM Python library. Recently, it got updated from "MLC Chat" to "MLC LLM".

Have you tried reinstalling them? python3 -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly mlc-ai-nightly

[RageshAntonyHM]

@MasterJH5574

I able to build the app and run it.

But when trying to run a finetuned model provided by @NSTiwari , I get this error:

jar libs:

libs 2.zip

[MasterJH5574]

Hello folks, sorry for the delayed response. Unfortunately I don't have enough bandwidth to build the android app. But you are more than welcome to coordinate here.

As for the issue @RageshAntonyHM you posted, I believe recompiling the model will address this issue. The issue is caused by a recent change in https://github.com/mlc-ai/mlc-llm/commit/9ecc00edd3cff739cfbd4ae781409d98060a8ac2, and recompiling the model helps resolve it.