Exception: Error generating stop tokens for geneach loop.

[Sam1320]

The bug

AssertionError: When calling OpenAI chat models you must generate only directly inside the assistant role! The OpenAI API does not currently support partial assistant prompting.
...
Exception: Error generating stop tokens for geneach loop. Perhaps you are outside of role tags (assistant/user/system)? If you don't want the loop to check for stop tokens, set stop=False or set num_iterations.

The error goes away after seting stop=False as suggested in the error message but I think this should be added to the readme. I can do the pull request if you agree.

To Reproduce I copy pasted the example Agents with geneach given in the readme.

import guidance
guidance.llm = guidance.llms.OpenAI('gpt-3.5-turbo')
guidance.llm.cache.clear()

prompt = guidance(
'''{{#system~}}
You are a helpful assistant
{{~/system}}
{{~#geneach 'conversation'}}
{{#user~}}
{{set 'this.user_text' (await 'user_text')}}
{{~/user}}
{{#assistant~}}
{{gen 'this.ai_text' temperature=0 max_tokens=300}}
{{~/assistant}}
{{~/geneach}}''')
prompt= prompt(user_text ='hi there')
prompt

System info (please complete the following information):

• OS: Ubuntu
• Guidance Version: 0.0.47 and 0.0.48

[boukeversteegh]

where did you set stop=False? could you provide the code where your workaround is applied? does it still allow you to execute the full guidance program? asking cause i have the same issue

[mcapodici]

I had a similar exception, but it didn't mention stop=False

To fix I had to make sure that my text was in a separate block from the gen:

Fails:

{{#assistant~}}
desserts:

{{gen 'desserts' n=10}}
{{~/assistant}}

Works:

{{#assistant~}}
desserts:

{{~/assistant}}
{{#assistant~}}
{{gen 'desserts' n=10}}
{{~/assistant}}

[Sam1320]

where did you set stop=False? could you provide the code where your workaround is applied? does it still allow you to execute the full guidance program? asking cause i have the same issue

Sorry for the late reply @boukeversteegh this is where I put the stop=False

{{~#geneach 'conversation' stop=False~}}

[marcotcr]

The error goes away after seting stop=False as suggested in the error message but I think this should be added to the readme. I can do the pull request if you agree.

Yes, we need to document this in the README. Will welcome a pull request, thank you :)

[ghaffarinia]

Probably unrelated to the bug, but where is a documentation I can find on how to use "stop" in geneach? I found two examples in https://github.com/guidance-ai/guidance/blob/main/notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb, but none of which explains it.

What I want to do is to make geneach stop with some token generated by assistant role.