llamafile
llamafile copied to clipboard
Unable to run on Windows: infinite loop with some messages
Running file of Windows give me numerous
$ llava-v1.5-7b-q4.exe
note: if you have an AMD or NVIDIA GPU then you need to pass -ngl 9999 to enable GPU offloading
note: if you have an AMD or NVIDIA GPU then you need to pass -ngl 9999 to enable GPU offloading
note: if you have an AMD or NVIDIA GPU then you need to pass -ngl 9999 to enable GPU offloading
note: if you have an AMD or NVIDIA GPU then you need to pass -ngl 9999 to enable GPU offloading
...
If I provide the flag it spams with another message
$ llava-v1.5-7b-q4.exe -ngl 9999
import_cuda_impl: initializing gpu module...
get_rocm_bin_path: note: amdclang++.exe not found on $PATH
get_rocm_bin_path: note: $HIP_PATH/bin/amdclang++.exe does not exist
get_rocm_bin_path: note: /opt/rocm/bin/amdclang++.exe does not exist
get_rocm_bin_path: note: clang++.exe not found on $PATH
get_rocm_bin_path: note: $HIP_PATH/bin/clang++.exe does not exist
get_rocm_bin_path: note: /opt/rocm/bin/clang++.exe does not exist
link_cuda_dso: note: dynamically linking C:\Users\Dmitry/.llamafile/ggml-rocm.dll
link_cuda_dso: warning: library not found: failed to load library
get_nvcc_path: note: nvcc.exe not found on $PATH
get_nvcc_path: note: $CUDA_PATH/bin/nvcc.exe does not exist
get_nvcc_path: note: /opt/cuda/bin/nvcc.exe does not exist
get_nvcc_path: note: /usr/local/cuda/bin/nvcc.exe does not exist
link_cuda_dso: note: dynamically linking C:\Users\Dmitry/.llamafile/ggml-cuda.dll
ggml_cuda_link: welcome to CUDA SDK with tinyBLAS
link_cuda_dso: GPU support linked
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce GTX 1070, compute capability 6.1, VMM: yes
link_cuda_dso: GPU support loaded
{"timestamp":1711288382,"level":"INFO","function":"server_cli","line":2457,"message":"build info","build":1500,"commit":"a30b324"}
{"timestamp":1711288382,"level":"INFO","function":"server_cli","line":2457,"message":"build info","build":1500,"commit":"a30b324"}
{"timestamp":1711288382,"level":"INFO","function":"server_cli","line":2457,"message":"build info","build":1500,"commit":"a30b324"}
{"timestamp":1711288382,"level":"INFO","function":"server_cli","line":2457,"message":"build info","build":1500,"commit":"a30b324"}
...
What can be done?