ollama
ollama copied to clipboard
Added logit_bias support
This PR brings the logit_bias functionality, already present in llama.cpp, which allows users to adjust the likelihood of token occurrences in generated text.
For example, for the prompt "Once upon a " we have:
-
Without
logit_biascurl -X POST http://localhost:11434/api/generate -d '{ "model": "llama2:7b", "prompt": "Once upon a ", "stream": false, "options": {"seed": 0, "num_predict" : 10} }' {"model":"llama2:7b", ... , "response":"time, in a far-off land, there", ...} -
With
logit_biasremoving "time" and "Time" words:curl -X POST http://localhost:11434/api/generate -d '{ "model": "llama2:7b", "prompt": "Once upon a ", "stream": false, "options": {"seed": 0, "num_predict" : 10, "logit_bias": [[2230,false],[931,false],[2481,false],[5974,false]]} }' {"model":"llama2:7b", ... , "response":"Once upon a...! What a great prompt!", ...} -
With
logit_biasremoving "time" and "Time" words, and increasing "night" likelihood :curl -X POST http://localhost:11434/api/generate -d '{ "model": "llama2:7b", "prompt": "Once upon a ", "stream": false, "options": {"seed": 0, "num_predict" : 10, "logit_bias": [[2230,false],[931,false],[2481,false],[5974,false],[11147, 15]]} }' {"model":"llama2:7b", ... , "response":"night, in a land far away, there lived"", ...}
Hi Voktruz, any chance of updating this for the latest main?
any updates on this?