[Bad Case]: 手机运行多模态失败

[RiverFerry]

Description / 描述

环境：

问题

使用这里提供的apk：https://github.com/OpenBMB/mlc-MiniCPM 安装后下载完模型，按照说明操作到了ready to chat后上传图片就没有反应了，没有出现预期的“image process is done” 输入文字会直接报错：

详细报错：

MLCChat failed

Stack trace:
org.apache.tvm.Base$TVMError: InternalError: Check failed: (chat_ != nullptr) is false: Chat is not initialized via reload
Stack trace:
  File "/Users/acha/Desktop/mlc-MiniCPM/cpp/[llm_chat.cc](http://llm_chat.cc/)", line 1698

	at org.apache.tvm.Base.checkCall(Base.java:173)
	at org.apache.tvm.Function.invoke(Function.java:130)
	at com.modelbest.mlcllm.ChatModule.prefill(ChatModule.java:55)
	at com.modelbest.minicpm.AppViewModel$ChatState$requestGenerate$1$2.invoke(AppViewModel.kt:651)
	at com.modelbest.minicpm.AppViewModel$ChatState$requestGenerate$1$2.invoke(AppViewModel.kt:651)
	at com.modelbest.minicpm.AppViewModel$ChatState.callBackend(AppViewModel.kt:515)
	at com.modelbest.minicpm.AppViewModel$ChatState.requestGenerate$lambda$5(AppViewModel.kt:651)
	at com.modelbest.minicpm.AppViewModel$ChatState.$r8$lambda$xrIi56HYcCEQD5s2-wW8Uqe1KbY(Unknown Source:0)
	at com.modelbest.minicpm.AppViewModel$ChatState$$ExternalSyntheticLambda0.run(Unknown Source:8)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:463)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
	at java.lang.Thread.run(Thread.java:1012)


Error message:
InternalError: Check failed: (chat_ != nullptr) is false: Chat is not initialized via reload
Stack trace:
  File "/Users/acha/Desktop/mlc-MiniCPM/cpp/[llm_chat.cc](http://llm_chat.cc/)", line 1698

Case Explaination / 案例解释

No response

[RiverFerry]

文本模型可以运行

[Achazwl]

可能和这个有关 https://github.com/mlc-ai/mlc-llm/issues/665 ，怀疑和模型无响应（例如手机内存不足）有关

[w-wu14]

链接的apk下载不了了吗？点积下载链接显示如下： <Error> <Code>UserDisable</Code> <Message>UserDisable</Message> <RequestId>65C18EB7FC567C3437887155</RequestId> <HostId>openbmb.oss-cn-hongkong.aliyuncs.com</HostId> <EC>0003-00000801</EC> <RecommendDoc>https://api.aliyun.com/troubleshoot?q=0003-00000801</RecommendDoc> </Error>

[RiverFerry]

可能和这个有关 mlc-ai/mlc-llm#665 ，怀疑和模型无响应（例如手机内存不足）有关

我更新了手机操作系统现在图片可以发送出去了，但是还是报错，这次的错误信息直接提示了内存不足，我的手机是12gb的内存，问下官方的demo演示用的是多大的内存。提供的apk是用的int4量化版本的模型吗。推荐的运行最低配置是多少，我现在的报错是：

MLCChat failed

Stack trace:
org.apache.tvm.Base$TVMError: TVMError: OpenCL build error for device=0x77c655bb58
Error: CL_OUT_OF_HOST_MEMORY
Pass
Stack trace:
  File "/Users/acha/Desktop/mlc-MiniCPM/3rdparty/tvm/src/runtime/[library_module.cc](http://library_module.cc/)", line 78

	at org.apache.tvm.Base.checkCall(Base.java:173)
	at org.apache.tvm.Function.invoke(Function.java:130)
	at com.modelbest.mlcllm.ChatModule.image(ChatModule.java:63)
	at com.modelbest.minicpm.AppViewModel$ChatState$requestImage$1$1.invoke(AppViewModel.kt:629)
	at com.modelbest.minicpm.AppViewModel$ChatState$requestImage$1$1.invoke(AppViewModel.kt:629)
	at com.modelbest.minicpm.AppViewModel$ChatState.callBackend(AppViewModel.kt:515)
	at com.modelbest.minicpm.AppViewModel$ChatState.requestImage$lambda$4(AppViewModel.kt:629)
	at com.modelbest.minicpm.AppViewModel$ChatState.$r8$lambda$pENZ-KlflB_gdrlr6E2VgmxXkJA(Unknown Source:0)
	at com.modelbest.minicpm.AppViewModel$ChatState$$ExternalSyntheticLambda2.run(Unknown Source:6)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
	at java.lang.Thread.run(Thread.java:1012)


Error message:
TVMError: OpenCL build error for device=0x77c655bb58
Error: CL_OUT_OF_HOST_MEMORY
Pass
Stack trace:
  File "/Users/acha/Desktop/mlc-MiniCPM/3rdparty/tvm/src/runtime/[library_module.cc](http://library_module.cc/)", line 78

后面我尝试了本地编译，到这一步报错了：

python -m mlc_chat convert_weight --model-type ${MODEL_TYPE} ../dist/models/${MODEL_NAME}/ --quantization $QUANTIZATION -o ../dist/$MODEL_NAME/
/Users/river/Library/Python/3.9/lib/python/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
[2024-02-06 01:14:00] INFO auto_config.py:115: Found model configuration: ../dist/models/MiniCPM-V/config.json
[2024-02-06 01:14:00] INFO auto_device.py:85: Not found device: cuda:0
[2024-02-06 01:14:01] INFO auto_device.py:85: Not found device: rocm:0
[2024-02-06 01:14:01] INFO auto_device.py:76: Found device: metal:0
[2024-02-06 01:14:01] INFO auto_device.py:85: Not found device: vulkan:0
[2024-02-06 01:14:02] INFO auto_device.py:85: Not found device: opencl:0
[2024-02-06 01:14:02] INFO auto_device.py:33: Using device: metal:0
[2024-02-06 01:14:02] INFO auto_weight.py:70: Finding weights in: ../dist/models/MiniCPM-V
[2024-02-06 01:14:02] INFO auto_weight.py:136: Not found Huggingface PyTorch
[2024-02-06 01:14:02] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: ../dist/models/MiniCPM-V/model.safetensors.index.json
[2024-02-06 01:14:02] INFO auto_weight.py:106: Using source weight configuration: ../dist/models/MiniCPM-V/model.safetensors.index.json. Use `--source` to override.
[2024-02-06 01:14:02] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use `--source-format` to override.
[2024-02-06 01:14:02] INFO auto_config.py:153: Found model type: minicpm_v. Use `--model-type` to override.
Weight conversion with arguments:
  --config          ../dist/models/MiniCPM-V/config.json
  --quantization    GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', linear_weight_layout='NK', num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
  --model-type      minicpm_v
  --device          metal:0
  --source          ../dist/models/MiniCPM-V/model.safetensors.index.json
  --source-format   huggingface-safetensor
  --output          ../dist/MiniCPM-V
[2024-02-06 01:14:02] INFO mistral_model.py:55: prefill_chunk_size defaults to sliding_window_size (4096)
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/__main__.py", line 39, in <module>
    main()
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/__main__.py", line 28, in main
    cli.main(sys.argv[2:])
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/cli/convert_weight.py", line 87, in main
    convert_weight(
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 156, in convert_weight
    _convert_args(args)
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 76, in _convert_args
    _, _named_params, _ = model.export_tvm(  # type: ignore[misc]
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/core.py", line 479, in export_tvm
    mod, params, ext_mods = Exporter(debug=debug).build(spec)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/exporter.py", line 136, in build
    outputs, inputs = _emit_method(self.builder, method_spec, params, effects)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/exporter.py", line 277, in _emit_method
    outputs = spec.method(*explicit_inputs)
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/model/mistral/mistral_model.py", line 700, in image
    inputs = self.vpm(inputs)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/core.py", line 427, in __call__
    return self.forward(*args, **kwargs)  # pylint: disable=no-member
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/model/mistral/vit_model.py", line 195, in forward
    hidden_states = self.patch_embed(inputs)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/core.py", line 427, in __call__
    return self.forward(*args, **kwargs)  # pylint: disable=no-member
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/model/mistral/vit_model.py", line 155, in forward
    embed = self.proj(inputs)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/core.py", line 427, in __call__
    return self.forward(*args, **kwargs)  # pylint: disable=no-member
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/subroutine.py", line 87, in new_forward
    return old_forward(self, *args, **kwargs)
  File "/Users/river/code/mlc-MiniCPM/python/mlc_chat/model/mistral/vit_model.py", line 141, in forward
    x = x + self.bias.reshape([1, self.bias.shape[0], 1, 1])
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/_tensor_op.py", line 82, in reshape
    return _op().reshape(self, shape)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/frontend/nn/op.py", line 644, in reshape
    return wrap_nested(_op.reshape(x._expr, shape), name)
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/relax/op/manipulate.py", line 215, in reshape
    return _ffi_api.reshape(x, shape)  # type: ignore
  File "tvm/_ffi/_cython/./packed_func.pxi", line 332, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 263, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 252, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 182, in tvm._ffi._cy3.core.CHECK_CALL
  File "/Users/river/Library/Python/3.9/lib/python/site-packages/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
tvm._ffi.base.TVMError: Traceback (most recent call last):
  File "/Users/catalyst/Workspace/mlc-ai-package-self-runner/_work/package/package/tvm/src/relax/op/tensor/[manipulate.cc](http://manipulate.cc/)", line 675
TVMError: Check failed: (_len != nullptr) is false: Reshape only expects the input new shape to be either an Expr or an Array of PrimExprs. However, the given new shape is [[1, 1152, 1, 1]]
[01:14:02] /Users/catalyst/Workspace/mlc-ai-package-self-runner/_work/package/package/tvm/src/relax/ir/[block_builder.cc:65](http://block_builder.cc:65/): Warning: BlockBuilder destroyed with remaining blocks!

[Achazwl]

我的手机配置是 oneplus 9R, 12GB 运行内存（没额外使用内存扩展），骁龙 870 八核。提供的 apk 是 int4 版本的。 我试了些同样是 oneplus 系列或 oppo 系列的新手机，可用运行内存更大，但是却不能运行，现在尚不明确具体原因。

[helloxcq]

链接的apk下载不了了吗？点积下载链接显示如下： UserDisable UserDisable 65C18EB7FC567C3437887155 openbmb.oss-cn-hongkong.aliyuncs.com 0003-00000801 https://api.aliyun.com/troubleshoot?q=0003-00000801

现在应该是可以下载了。

[RiverFerry]

我的手机配置是 oneplus 9R, 12GB 运行内存（没额外使用内存扩展），骁龙 870 八核。提供的 apk 是 int4 版本的。 我试了些同样是 oneplus 系列或 oppo 系列的新手机，可用运行内存更大，但是却不能运行，现在尚不明确具体原因。

请问现在多模态的有进展吗？还有上面我贴的自己编译遇到的问题麻烦给看下： File "/Users/catalyst/Workspace/mlc-ai-package-self-runner/_work/package/package/tvm/src/relax/op/tensor/manipulate.cc", line 675 TVMError: Check failed: (_len != nullptr) is false: Reshape only expects the input new shape to be either an Expr or an Array of PrimExprs. However, the given new shape is [[1, 1152, 1, 1]] [01:14:02] /Users/catalyst/Workspace/mlc-ai-package-self-runner/_work/package/package/tvm/src/relax/ir/block_builder.cc:65: Warning: BlockBuilder destroyed with remaining blocks!

[Achazwl]

多模态现在还在研究。你这个 shape [[1, 1152, 1, 1]] 的问题应该和 #47 是一样的，是版本问题