Select function always selects the same option for a specific prompt.

[Haseeb-BSAA]

I'm new to LLMs, so I'm sorry if I'm missing something obvious. However, when I run this simple code: from guidance import models, gen, select, user, assistant, system, substring model_path = "Llama-3.2-3B-Instruct-Q4_K_M.gguf" llm = models.LlamaCpp(model_path, n_gpu_layers=-1) llm + f'Pick a random number: ' + select(['One',"Two","Three","Four",'Five',"Six","Seven","Eight","Nine","Ten"])

I always get the same output, which is Eight. If I change the prompt a tiny bit, for example if I remove the colon and the space, it would always get Nine. What's wrong?

I also tried running the example code on guidance's main github page: llama2 + f'Do you want a joke or a poem? A ' + select(['joke', 'poem']) Same thing here, it always picks the same option.

I tried changing top_k and temperature parameters when loading the model, but nothing seems to work.

[BorisTheBrave]

Yes, I have the same problem.

I note that with_temperature doesn't work either.

def run():
    model = "gpt2"
    lm = models.Transformers(model)

    print(lm + 'I like the color ' + select(['red', 'blue', 'green'])) # always blue
    print(lm + 'I like the color ' + with_temperature(select(['red', 'blue', 'green']), 10))  # always blue

[BorisTheBrave]

Ok, this link suggests that guidance is always greedy. So temperature doesn't do anything in any case!

What a pain - you've given some subtle control over generation, but taken away more powerful options.