mlc-llm

mlc-llm copied to clipboard

Does this not support AMD GPUs?

I'm getting this error:

terminate called after throwing an instance of 'tvm::runtime::InternalError'
  what():  [18:53:29] /home/runner/work/utils/utils/tvm/src/runtime/vulkan/vulkan_instance.cc:111: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (__e == VK_SUCCESS) is false: Vulkan Error, code=-9: VK_ERROR_INCOMPATIBLE_DRIVER
Stack trace:
  [bt] (0) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(tvm::runtime::Backtrace[abi:cxx11]()+0x27) [0x7f6fa5518a37]
  [bt] (1) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(+0x3f375) [0x7f6fa54b6375]
  [bt] (2) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(tvm::runtime::vulkan::VulkanInstance::VulkanInstance()+0x1a47) [0x7f6fa5605857]
  [bt] (3) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(tvm::runtime::vulkan::VulkanDeviceAPI::VulkanDeviceAPI()+0x40) [0x7f6fa5601a60]
  [bt] (4) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(tvm::runtime::vulkan::VulkanDeviceAPI::Global()+0x4c) [0x7f6fa5601dfc]
  [bt] (5) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(+0x18ae3d) [0x7f6fa5601e3d]
  [bt] (6) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(+0x6bdc4) [0x7f6fa54e2dc4]
  [bt] (7) /home/loganm/miniconda3/envs/llama_index/bin/../lib/libtvm_runtime.so(+0x6c367) [0x7f6fa54e3367]
  [bt] (8) mlc_chat_cli(+0xd950) [0x55ed543c8950]


Aborted

Here's my driver info

Yes, i think there are ppl successfully running on AMD GPUs, https://www.reddit.com/r/LocalLLaMA/comments/132igcy/project_mlc_llm_universal_llm_deployment_with_gpu/ please try to upgrade to the latest driver and see if it works

the same error

~# mlc_chat_cli
terminate called after throwing an instance of 'tvm::runtime::InternalError'
  what():  [21:35:46] /home/runner/work/utils/utils/tvm/src/runtime/vulkan/vulkan_instance.cc:111:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (__e == VK_SUCCESS) is false: Vulkan Error, code=-9: VK_ERROR_INCOMPATIBLE_DRIVER
Stack trace:
  [bt] (0) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::Backtrace[abi:cxx11]()+0x27) [0x7f4190835a37]
  [bt] (1) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x3f375) [0x7f41907d3375]
  [bt] (2) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::vulkan::VulkanInstance::VulkanInstance()+0x1a47) [0x7f4190922857]
  [bt] (3) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::vulkan::VulkanDeviceAPI::VulkanDeviceAPI()+0x40) [0x7f419091ea60]
  [bt] (4) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::vulkan::VulkanDeviceAPI::Global()+0x4c) [0x7f419091edfc]
  [bt] (5) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x18ae3d) [0x7f419091ee3d]
  [bt] (6) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x6bdc4) [0x7f41907ffdc4]
  [bt] (7) /root/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x6c367) [0x7f4190800367]
  [bt] (8) mlc_chat_cli(+0xe010) [0x5654d12ee010]

gpu info

Yea I've updated all my drivers. I wish there was an easier way to debug haha

@logan-markewich does it work now with your AMD GPU? If so, would you mind sharing your tok/sec info here?

@junrushao I should have clarified, I updated everything but couldn't get it to work haha

Not sure how to debug it either

I just bought an AMD GPU online. It's arriving later this week. Will play with it and see how it works with the Vulkan driver.

should be resolved in latest instructions https://mlc.ai/mlc-llm/docs/