Is it possible to run llama.cpp in Google Colab Pro?

[codepunisher10]

Any help or guidance would be greatly appreciated.

[freedomtan]

upload models. let the directory be model

!git clone https://github.com/ggerganov/llama.cpp.git
!(cd llama.cpp; make)
!llama.cpp/main -m ..... -t 4 -l 128 .....

[PhilDakin]

Alternatively use https://github.com/abetlen/llama-cpp-python. This will handle caching the model.

!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python

llm = Llama(
  model_path="",
  n_gpu_layers=...,
  n_ctx=...,
)