kitops

kitops copied to clipboard

Describe the problem you're trying to solve dev mode should be able to load lora-adapters if they are present in the modelkit

Describe the solution you'd like dev mode should detect if there is a lora-adapter type on the modelkit and load it to the dev-mode.

Are we looking for something like model.parts.type and adding those in dev mode.

Does the llamafile we are downloading support --lora tag to run lora adapters?

Yes it does. Here are the related options. They are just inherited from llama.cpp

--lora FNAME
Apply LoRA adapter (implies --no-mmap)
--lora-scaled FNAME S
Apply LoRA adapter with user defined scaling S (implies --no-mmap)
--lora-base FNAME
Optional model to use as a base for the layers modified by the LoRA adapter