ollama icon indicating copy to clipboard operation
ollama copied to clipboard

Added logit_bias support

Open Vokturz opened this issue 2 years ago • 2 comments

This PR brings the logit_bias functionality, already present in llama.cpp, which allows users to adjust the likelihood of token occurrences in generated text.

For example, for the prompt "Once upon a " we have:

  1. Without logit_bias

    curl -X POST http://localhost:11434/api/generate -d '{
       "model": "llama2:7b",
       "prompt": "Once upon a ",
       "stream": false,
       "options": {"seed": 0, "num_predict" : 10}
       }'
    {"model":"llama2:7b", ... , "response":"time, in a far-off land, there", ...}
    
  2. With logit_bias removing "time" and "Time" words:

    curl -X POST http://localhost:11434/api/generate -d '{
       "model": "llama2:7b",
       "prompt": "Once upon a ",
       "stream": false,
       "options": {"seed": 0, "num_predict" : 10,
                   "logit_bias": [[2230,false],[931,false],[2481,false],[5974,false]]}
       }'
    {"model":"llama2:7b", ... , "response":"Once upon a...! What a great prompt!", ...}
    
  3. With logit_bias removing "time" and "Time" words, and increasing "night" likelihood :

    curl -X POST http://localhost:11434/api/generate -d '{
       "model": "llama2:7b",
       "prompt": "Once upon a ",
       "stream": false,
       "options": {"seed": 0, "num_predict" : 10,
                   "logit_bias": [[2230,false],[931,false],[2481,false],[5974,false],[11147, 15]]}
       }'
    {"model":"llama2:7b", ... , "response":"night, in a land far away, there lived"", ...}
    

Vokturz avatar Nov 07 '23 14:11 Vokturz

Hi Voktruz, any chance of updating this for the latest main?

nathanpbell avatar Jan 31 '24 01:01 nathanpbell

any updates on this?

K0IN avatar Mar 16 '24 16:03 K0IN