ImportError: DLL load failed while importing exllama_ext: 找不到指定的模块。

[onexixi]

exllama_ext = load( name = extension_name, sources = [ os.path.join(library_dir, "exllama_ext/exllama_ext.cpp"), os.path.join(library_dir, "exllama_ext/cuda_buffers.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/q4_matrix.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/q4_matmul.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/column_remap.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/rms_norm.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/rope.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/half_matmul.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/q4_attn.cu"), os.path.join(library_dir, "exllama_ext/cuda_func/q4_mlp.cu"), os.path.join(library_dir, "exllama_ext/cpu_func/rep_penalty.cpp") ], extra_include_paths = [os.path.join(library_dir, "exllama_ext")], verbose = verbose, extra_ldflags = (["cublas.lib"] + ([f"/LIBPATH:{os.path.join(sys.base_prefix, 'libs')}"] if sys.base_prefix != sys.prefix else [])) if windows else [], extra_cuda_cflags = ["-lineinfo"] + (["-U__HIP_NO_HALF_CONVERSIONS__", "-O3"] if torch.version.hip else []), extra_cflags = ["-O3"] # extra_cflags = ["-ftime-report", "-DTORCH_USE_CUDA_DSA"] )

When I use this, it reported an error

[onexixi]

Windows text-generation-webui (venv) python 310

[turboderp]

Any clue which DLL it's failing to find?

[hobpond]

I had the same error. What resolved it for me was updating my environment to use pytorch-cuda=11.8 and downloaded https://developer.nvidia.com/cuda-11-8-0-download-archive.

Seems like my pytorch-cuda for my environment was not compatible.

conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia

[xhinker]

solved the problem, use nvcc --version to find out your cuda version, if your cuda is 12.1.

Then install the 12.1 torch.

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121

[EyeDeck]

Sounds about right. If PyTorch and CUDA toolkit versions are mismatched, it'll usually still successfully compile (on Windows, into exllama_ext.pyd, which is a DLL by another extension), but the DLL will just simply refuse to load.

[onexixi]

I'll try it. Thank you very much.

[onexixi]

It's because the version doesn't match，It work ，Thank