llama.cpp
llama.cpp copied to clipboard
Feature Request: Mapping model name to LoRA config
Prerequisites
- [X] I am running the latest code. Mention the version if possible as well.
- [X] I carefully followed the README.md.
- [X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- [X] I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
I came across this idea while working on #10994
The idea is that we can maintain a list of model name mapped to LoRA config, for example:
{
"llama-base": [{"id": 0, "scale": 0.0}, {"id": 1, "scale": 0.0}],
"llama-story": [{"id": 0, "scale": 1.0}, {"id": 1, "scale": 0.0}],
"llama-abliteration": [{"id": 0, "scale": 0.0}, {"id": 1, "scale": 1.0}],
"llama-story-abliteration": [{"id": 0, "scale": 0.5}, {"id": 1, "scale": 0.5}]
}
Then, user can switch the model by specifying model
in the request, for example:
# first user:
{
"model": "llama-story-abliteration",
"messages": [
{"role": "user", "content": "Write a NSFW story"}
]
}
# second user:
{
"model": "llama-base",
"messages": [
{"role": "user", "content": "Is this NSFW?"}
]
}
Motivation
N/A
Possible Implementation
No response