Vicuna-7B returns incorrect choice

[AndyJZhao]

The bug Hi, thanks for the effort of developing this awesome library. When I tried to use the select function on my local Vicuna-7B, the function is not working as expected, similar to #17. However, there are two issues:

1. The selection content is not hidden.
2. Generated content is out of choices.

To Reproduce

# Modified the official guide to load Vicuna-7B
vicuna_path = '/path/to/my/checkpoints/vicuna-7b/'
llm = guidance.llms.transformers.Vicuna(vicuna_path, device_map="auto")

program = guidance('''Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe".
Sentence: {{example}}
Answer:{{#select "answer" logprobs='logprobs'}} Yes{{or}} No{{or}} Maybe{{/select}}''', llm=llm)
executed_program = program(example='I hate tacos')

The returned result:

Is the following sentence offensive? Please answer with a single word, either "Yes", "No", or "Maybe".
Sentence: I hate tacos
Answer: Yes No Maybee Yes

Issue 1: The selection content is not hidden. We can see that instead of generate the choice only, the choices are also returned.

Issue 2: Generated content is out of choices.

As shown in figure above, 'e Yes' is not a valid choice.

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Linux
• Guidance Version (guidance.__version__): 0.0.63