shao-shuai

> So this means no layers were put on gpu, but at least it recognized the gpu now. > > ```shell > llama_model_load_internal: offloading 0 repeating layers to GPU >...

> Yeah, it's not going to affect localGPT. But at least we know the underlying library works! You can try opening the text file and adding more layers, as long...

> If you haven't, can you try running this again? I believe the webui script runs on a separate conda env. > > ```shell > CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install -U...

> Perhaps llama-cpp requires cuda 11 or 12, but I couldn't find that in their documentation and I wonder if it can be replicated. > > But, the out of...

> Reference in new issue Thanks, sorry to have a new error, :sweat: code ```Python if model_basename is not None: if ".ggml" in model_basename: logging.info("Using Llamacpp for GGML quantized models")...

> Hi @shao-shuai, were you able to resolve? sorry, caught by a flu, will let you know.

> pytorch nightly 12.1 I installed pytorch nightly 12.1 ```shell pip list | grep torch pytorch-triton 2.1.0+e6216047b8 torch 2.1.0.dev20230830+cu121 torchaudio 2.1.0.dev20230830+cu121 torchvision 0.16.0.dev20230830+cu121 ``` still got the mismatch error ```shell...

> Hi, hope you are feeling better! Can you try to install cuda 12.1? sure, thanks, much better now. I tried cuda 12.1, still same error ```shell nvcc -V nvcc:...

> Did you reinstall llama-cpp? should I run this? ```shell CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir ```

> yes! updated llama-cpp, but can't load model this time:disappointed_relieved: ```shell python run_localGPT.py 2023-09-01 08:45:14,148 - INFO - run_localGPT.py:180 - Running on: cuda 2023-09-01 08:45:14,148 - INFO - run_localGPT.py:181 -...