Alpaca icon indicating copy to clipboard operation
Alpaca copied to clipboard
verified publisher icon Jeffser

Missing RX 5700 XT Support or bug with multiple GPU's

Open arne-fuchs opened this issue 8 months ago • 10 comments

ollama is able to execute LLMs on the Rx 5700 XT if you install it natively (Even if it is not officially supported in the GPU list, but rocm already supports it, so does ollama).

Alpaca with it's oolama instance is not able to detect the RX 5700 XT.

Here is a log at startup with debug of oolama enabled:

/app/lib/python3.12/site-packages/pydbus/registration.py:130: DeprecationWarning: Gio.DBusConnection.register_object is deprecated
  ids = [bus.con.register_object(path, interface, wrapper.call_method, None, None) for interface in interfaces]
INFO	[main.py | main] Alpaca version: 6.0.5
INFO	[instance_manager.py | start] Starting Alpaca's Ollama instance...
INFO	[instance_manager.py | start] Started Alpaca's Ollama instance
Couldn't find '/home/arne/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBInypp/tURGtO9Cnzbahh658XpUXX/7c5bZzY4i1Ygh

2025/05/07 16:52:20 routes.go:1233: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES:1 HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11435 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/arne/.var/app/com.jeffser.Alpaca/data/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:1 http_proxy: https_proxy: no_proxy:]"
time=2025-05-07T16:52:20.934+02:00 level=INFO source=images.go:458 msg="total blobs: 9"
time=2025-05-07T16:52:20.934+02:00 level=INFO source=images.go:465 msg="total unused blobs removed: 0"
time=2025-05-07T16:52:20.934+02:00 level=INFO source=routes.go:1300 msg="Listening on [::]:11435 (version 0.6.7)"
time=2025-05-07T16:52:20.934+02:00 level=DEBUG source=sched.go:107 msg="starting llm scheduler"
time=2025-05-07T16:52:20.934+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-05-07T16:52:20.936+02:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-05-07T16:52:20.936+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-05-07T16:52:20.936+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/app/plugins/Ollama/lib/ollama/libcuda.so* /app/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/sdk/llvm15/lib/libcuda.so* /app/plugins/lib/libcuda.so* /app/plugins/lib/ollama/libcuda.so* /app/plugins/lib/ollama/rocm/libcuda.so* /app/plugins/lib/ollama/rocm/rocblas/library/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
INFO	[instance_manager.py | start] client version is 0.6.7
time=2025-05-07T16:52:20.939+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[]
time=2025-05-07T16:52:20.939+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so*
time=2025-05-07T16:52:20.939+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/app/plugins/Ollama/lib/ollama/libcudart.so* /app/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcudart.so* /usr/lib/sdk/llvm15/lib/libcudart.so* /app/plugins/lib/libcudart.so* /app/plugins/lib/ollama/libcudart.so* /app/plugins/lib/ollama/rocm/libcudart.so* /app/plugins/lib/ollama/rocm/rocblas/library/libcudart.so* /app/plugins/Ollama/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/app/plugins/Ollama/lib/ollama/cuda_v11/libcudart.so.11.3.109 /app/plugins/Ollama/lib/ollama/cuda_v12/libcudart.so.12.8.90]"
cudaSetDevice err: 35
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/plugins/Ollama/lib/ollama/cuda_v11/libcudart.so.11.3.109: your nvidia driver is too old or missing.  If you have a CUDA GPU please upgrade to run ollama"
cudaSetDevice err: 35
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/plugins/Ollama/lib/ollama/cuda_v12/libcudart.so.12.8.90: your nvidia driver is too old or missing.  If you have a CUDA GPU please upgrade to run ollama"
time=2025-05-07T16:52:20.941+02:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=amd_linux.go:101 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=amd_linux.go:121 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=amd_linux.go:101 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties"
time=2025-05-07T16:52:20.941+02:00 level=DEBUG source=amd_linux.go:206 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29471 unique_id=0
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-DP-1/device/vendor error="open /sys/class/drm/card1-DP-1/device/vendor: no such file or directory"
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-DP-2/device/vendor error="open /sys/class/drm/card1-DP-2/device/vendor: no such file or directory"
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-DVI-D-1/device/vendor error="open /sys/class/drm/card1-DVI-D-1/device/vendor: no such file or directory"
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-HDMI-A-1/device/vendor error="open /sys/class/drm/card1-HDMI-A-1/device/vendor: no such file or directory"
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-HDMI-A-2/device/vendor error="open /sys/class/drm/card1-HDMI-A-2/device/vendor: no such file or directory"
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:240 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card2/device
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:318 msg="amdgpu memory" gpu=0 total="8.0 GiB"
time=2025-05-07T16:52:20.942+02:00 level=DEBUG source=amd_linux.go:319 msg="amdgpu memory" gpu=0 available="6.0 GiB"
time=2025-05-07T16:52:20.942+02:00 level=INFO source=amd_linux.go:332 msg="filtering out device per user request" id=0 visible_devices=[1]
time=2025-05-07T16:52:20.942+02:00 level=INFO source=amd_linux.go:402 msg="no compatible amdgpu devices detected"
time=2025-05-07T16:52:20.942+02:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-05-07T16:52:20.942+02:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="31.2 GiB" available="18.0 GiB"
[GIN] 2025/05/07 - 16:52:20 | 200 |     361.919µs |       127.0.0.1 | GET      "/api/tags"
time=2025-05-07T16:52:20.959+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-05-07T16:52:20.960+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-05-07T16:52:20.973+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-05-07T16:52:20.973+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
[GIN] 2025/05/07 - 16:52:20 | 200 |   27.941987ms |       127.0.0.1 | POST     "/api/show"
[GIN] 2025/05/07 - 16:52:20 | 200 |   27.146413ms |       127.0.0.1 | POST     "/api/show"
INFO	[window.py | closing_app] Closing app...
INFO	[instance_manager.py | stop] Stopping Alpaca's Ollama instance
INFO	[instance_manager.py | stop] Stopped Alpaca's Ollama instance

Installing Ollama and connecting to it via Alpaca works with RX 5700 XT acc.

arne-fuchs avatar May 07 '25 15:05 arne-fuchs

This line looks suspicious:

time=2025-05-07T16:52:20.942+02:00 level=INFO source=amd_linux.go:332 msg="filtering out device per user request" id=0 visible_devices=[1]

Did you specify ROCM_VISIBLE_DEVICES in your instance settings, and can you try setting it to 0?

mags0ft avatar May 07 '25 17:05 mags0ft

Here, I've set it to 0:

debian@pop-os:~$ ROCR_VISIBLE_DEVICES=0 OLLAMA_DEBUG=1 flatpak run com.jeffser.Alpaca
/app/lib/python3.12/site-packages/pydbus/registration.py:130: DeprecationWarning: Gio.DBusConnection.register_object is deprecated
  ids = [bus.con.register_object(path, interface, wrapper.call_method, None, None) for interface in interfaces]
INFO	[main.py | main] Alpaca version: 6.0.5
INFO	[instance_manager.py | start] Starting Alpaca's Ollama instance...
INFO	[instance_manager.py | start] Started Alpaca's Ollama instance
Couldn't find '/home/arne/.ollama/id_ed25519'. Generating new private key.
Your new public key is: 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDHhXz03K+u+zFBXoFY1PGTX05uggmVFj1/DNoGShwb8

2025/05/07 19:36:01 routes.go:1233: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES:1 HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11435 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/arne/.var/app/com.jeffser.Alpaca/data/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES:1 http_proxy: https_proxy: no_proxy:]"
time=2025-05-07T19:36:01.102+02:00 level=INFO source=images.go:458 msg="total blobs: 9"
time=2025-05-07T19:36:01.102+02:00 level=INFO source=images.go:465 msg="total unused blobs removed: 0"
time=2025-05-07T19:36:01.102+02:00 level=INFO source=routes.go:1300 msg="Listening on [::]:11435 (version 0.6.7)"
time=2025-05-07T19:36:01.103+02:00 level=DEBUG source=sched.go:107 msg="starting llm scheduler"
time=2025-05-07T19:36:01.103+02:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
INFO	[instance_manager.py | start] client version is 0.6.7
time=2025-05-07T19:36:01.105+02:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-05-07T19:36:01.105+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-05-07T19:36:01.105+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/app/plugins/Ollama/lib/ollama/libcuda.so* /app/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/sdk/llvm15/lib/libcuda.so* /app/plugins/lib/libcuda.so* /app/plugins/lib/ollama/libcuda.so* /app/plugins/lib/ollama/rocm/libcuda.so* /app/plugins/lib/ollama/rocm/rocblas/library/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-05-07T19:36:01.108+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[]
time=2025-05-07T19:36:01.108+02:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so*
time=2025-05-07T19:36:01.108+02:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/app/plugins/Ollama/lib/ollama/libcudart.so* /app/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcudart.so* /usr/lib/sdk/llvm15/lib/libcudart.so* /app/plugins/lib/libcudart.so* /app/plugins/lib/ollama/libcudart.so* /app/plugins/lib/ollama/rocm/libcudart.so* /app/plugins/lib/ollama/rocm/rocblas/library/libcudart.so* /app/plugins/Ollama/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2025-05-07T19:36:01.110+02:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/app/plugins/Ollama/lib/ollama/cuda_v11/libcudart.so.11.3.109 /app/plugins/Ollama/lib/ollama/cuda_v12/libcudart.so.12.8.90]"
cudaSetDevice err: 35
time=2025-05-07T19:36:01.110+02:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/plugins/Ollama/lib/ollama/cuda_v11/libcudart.so.11.3.109: your nvidia driver is too old or missing.  If you have a CUDA GPU please upgrade to run ollama"
cudaSetDevice err: 35
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/plugins/Ollama/lib/ollama/cuda_v12/libcudart.so.12.8.90: your nvidia driver is too old or missing.  If you have a CUDA GPU please upgrade to run ollama"
time=2025-05-07T19:36:01.111+02:00 level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:101 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:121 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:101 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:206 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29471 unique_id=0
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-DP-1/device/vendor error="open /sys/class/drm/card1-DP-1/device/vendor: no such file or directory"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-DP-2/device/vendor error="open /sys/class/drm/card1-DP-2/device/vendor: no such file or directory"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-DVI-D-1/device/vendor error="open /sys/class/drm/card1-DVI-D-1/device/vendor: no such file or directory"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-HDMI-A-1/device/vendor error="open /sys/class/drm/card1-HDMI-A-1/device/vendor: no such file or directory"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:219 msg="failed to read sysfs node" file=/sys/class/drm/card1-HDMI-A-2/device/vendor error="open /sys/class/drm/card1-HDMI-A-2/device/vendor: no such file or directory"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:240 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card2/device
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:318 msg="amdgpu memory" gpu=0 total="8.0 GiB"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:319 msg="amdgpu memory" gpu=0 available="6.5 GiB"
time=2025-05-07T19:36:01.111+02:00 level=INFO source=amd_linux.go:332 msg="filtering out device per user request" id=0 visible_devices=[1]
time=2025-05-07T19:36:01.111+02:00 level=INFO source=amd_linux.go:402 msg="no compatible amdgpu devices detected"
time=2025-05-07T19:36:01.111+02:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-05-07T19:36:01.111+02:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="31.2 GiB" available="16.8 GiB"
[GIN] 2025/05/07 - 19:36:01 | 200 |     358.603µs |       127.0.0.1 | GET      "/api/tags"
time=2025-05-07T19:36:01.129+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-05-07T19:36:01.130+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
time=2025-05-07T19:36:01.143+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
[GIN] 2025/05/07 - 19:36:01 | 200 |    28.47016ms |       127.0.0.1 | POST     "/api/show"
time=2025-05-07T19:36:01.144+02:00 level=WARN source=ggml.go:152 msg="key not found" key=general.alignment default=32
[GIN] 2025/05/07 - 19:36:01 | 200 |   29.214596ms |       127.0.0.1 | POST     "/api/show"

arne-fuchs avatar May 07 '25 17:05 arne-fuchs

I may have to mention, that there is a second card, a RX 590, in the system.

arne-fuchs avatar May 07 '25 17:05 arne-fuchs

I may have to mention, that there is a second card, a RX 590, in the system.

Okay, that's good to know! Thanks for sending the new output.

The RX 590 is definitely not supported for Ollama inference, so if Alpaca makes Ollama / ROCm only see that, this might be causing the issues.

Could you please tell me the values for the overrides used for your integrated Ollama instance?

Image of the override inputs in Alpaca's instance manager

You may have more luck by clearing all the values. And don't forget to scroll down and hit "Save"! Another thing to try might be setting HIP_VISIBLE_DEVICES to 0 if the first tip doesn't help.

mags0ft avatar May 07 '25 17:05 mags0ft

Image

Setting HIP_VISIBLE_DEVICES to 0 made no difference as well as trying to set HSSAS_OVERRIDE_GFX_VERSION to 10.1.0 and ROCR_VISIBLE_DEVICES to 0

arne-fuchs avatar May 07 '25 18:05 arne-fuchs

That's actually kind of weird, as your logs are also saying

time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:318 msg="amdgpu memory" gpu=0 total="8.0 GiB"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:319 msg="amdgpu memory" gpu=0 available="6.5 GiB"

So apparently an AMD GPU is detected, but it's deliberately not used by Ollama.

I'm not entirely sure what the issue might be, but maybe someone else may be of better help.

mags0ft avatar May 07 '25 19:05 mags0ft

If I could help you with any data, please tell me

arne-fuchs avatar May 07 '25 20:05 arne-fuchs

That's actually kind of weird, as your logs are also saying

time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:318 msg="amdgpu memory" gpu=0 total="8.0 GiB"
time=2025-05-07T19:36:01.111+02:00 level=DEBUG source=amd_linux.go:319 msg="amdgpu memory" gpu=0 available="6.5 GiB"

So apparently an AMD GPU is detected, but it's deliberately not used by Ollama.

I'm not entirely sure what the issue might be, but maybe someone else may be of better help.

And this is the RX 5700 XT. It is the one, which is connected to the display, so this is why only 6.B GiB are abailable

arne-fuchs avatar May 07 '25 20:05 arne-fuchs

I had to unset HIP_VISIBLE_DEVICES for my RX 5600 XT to be detected and used (or so I thought, see below).

When ollama is about to use the GPU it logged

time=2025-05-13T10:40:48.299+02:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=rocm variant="" compute=gfx1010 driver=0.0 name=1002:731f total="6.0 GiB" available="4.7 GiB"

but later

time=2025-05-13T10:41:19.514+02:00 level=INFO source=sched.go:723 msg="new model will fit in available VRAM in single GPU, loading" model=/home/rene/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff gpu=0 parallel=2 available=5048242176 required="3.7 GiB"
...
time=2025-05-13T10:41:19.823+02:00 level=INFO source=server.go:409 msg="starting llama server" cmd="/app/plugins/Ollama/bin/ollama runner --model /home/rene/.var/app/com.jeffser.Alpaca/data/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff --ctx-size 8192 --batch-size 512 --n-gpu-layers 29 --threads 6 --parallel 2 --port 34961"
time=2025-05-13T10:41:19.824+02:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
time=2025-05-13T10:41:19.824+02:00 level=INFO source=server.go:585 msg="waiting for llama runner to start responding"
time=2025-05-13T10:41:19.824+02:00 level=INFO source=server.go:619 msg="waiting for server to become available" status="llm server error"
time=2025-05-13T10:41:19.835+02:00 level=INFO source=runner.go:853 msg="starting go runner"
load_backend: loaded CPU backend from /app/plugins/Ollama/lib/ollama/libggml-cpu-haswell.so
time=2025-05-13T10:41:19.840+02:00 level=INFO source=ggml.go:103 msg=system CPU.0.SSE3=1 CPU.0.SSSE3=1 CPU.0.AVX=1 CPU.0.AVX2=1 CPU.0.F16C=1 CPU.0.FMA=1 CPU.0.BMI2=1 CPU.0.LLAMAFILE=1 CPU.1.LLAMAFILE=1 compiler=cgo(gcc)

But no matter how small the model, Alpaca's Ollama always runs on the CPU, not on the GPU. "Standard" Ollama runs fine up to ~8B params on this GPU. I fear msg="waiting for server to become available" status="llm server error" may be what's then forcing the model on the CPU? Would be interested to hear what you end up with.

rkraneis avatar May 13 '25 09:05 rkraneis

I'm seeing the same as above and just also discovered that the native Arch package ollama-rocm works fine. It was actually hard to find this information because the AMD discussion is currently fragmented across several issues and a discussion (https://github.com/Jeffser/Alpaca/issues/770, https://github.com/Jeffser/Alpaca/issues/675, https://github.com/Jeffser/Alpaca/issues/603, https://github.com/Jeffser/Alpaca/issues/587, discussion https://github.com/Jeffser/Alpaca/discussions/582).

I don't know if this is useful or new info, but I also found these upstream troubleshooting steps in my searching: https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#amd-gpu-discovery

Adding those env vars and looking for differences between running ollama from the flatpak extension vs the native Arch ollama-rocm package, I only see that both detect my gpu and seem to try to use it but then the flatpak decides not to with level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible=[]

Flatpak:

time=2025-06-26T14:24:30.774-04:00 level=DEBUG source=memory.go:111 msg=evaluating library=rocm gpu_count=1 available="[5.6 GiB]"
time=2025-06-26T14:24:30.774-04:00 level=DEBUG source=ggml.go:155 msg="key not found" key=llama.vision.block_count default=0
time=2025-06-26T14:24:30.774-04:00 level=DEBUG source=ggml.go:155 msg="key not found" key=llama.attention.key_length default=128
time=2025-06-26T14:24:30.775-04:00 level=DEBUG source=ggml.go:155 msg="key not found" key=llama.attention.value_length default=128
time=2025-06-26T14:24:30.775-04:00 level=INFO source=server.go:168 msg=offload library=rocm layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[5.6 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.1 GiB" memory.required.partial="5.1 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[5.1 GiB]" memory.weights.total="3.8 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="105.0 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="305.0 MiB"
time=2025-06-26T14:24:30.775-04:00 level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible=[]
...
time=2025-06-26T14:24:30.825-04:00 level=DEBUG source=server.go:367 msg="adding gpu dependency paths" paths=[/app/plugins/lib/ollama/rocm]

Native:

time=2025-06-26T14:33:01.174-04:00 level=DEBUG source=memory.go:111 msg=evaluating library=rocm gpu_count=1 available="[5.5 GiB]"
time=2025-06-26T14:33:01.174-04:00 level=DEBUG source=ggml.go:155 msg="key not found" key=llama.vision.block_count default=0
time=2025-06-26T14:33:01.174-04:00 level=DEBUG source=ggml.go:155 msg="key not found" key=llama.attention.key_length default=128
time=2025-06-26T14:33:01.174-04:00 level=DEBUG source=ggml.go:155 msg="key not found" key=llama.attention.value_length default=128
time=2025-06-26T14:33:01.174-04:00 level=INFO source=server.go:168 msg=offload library=rocm layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[5.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.1 GiB" memory.required.partial="5.1 GiB" memory.required.kv="512.0 MiB" memory.required.allocations="[5.1 GiB]" memory.weights.total="3.8 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="105.0 MiB" memory.graph.full="296.0 MiB" memory.graph.partial="305.0 MiB"
time=2025-06-26T14:33:01.174-04:00 level=DEBUG source=server.go:284 msg="compatible gpu libraries" compatible=[rocm]
...
time=2025-06-26T14:33:01.220-04:00 level=DEBUG source=server.go:360 msg="adding gpu library" path=/usr/lib/ollama/rocm
time=2025-06-26T14:33:01.220-04:00 level=DEBUG source=server.go:367 msg="adding gpu dependency paths" paths=[/opt/rocm/lib]

inthreedee avatar Jun 26 '25 18:06 inthreedee