microsoft
Guaranteeing valid syntax JSON fails for arrays of numbers
The bug Although producing individual fields as numbers seems to work fine (although invariably the actual parsed variable is still a string as seen below), and arrays of strings also seem to work fine, arrays of numbers do not seem work properly.
I'd love to be able to fully understand why this is happening and am also very happy to attempt to fix this if someone is able to point me in the right direction.
To Reproduce
import guidance
import os
os.environ["OPENAI_API_KEY"] = "sk-..."
llm = guidance.llms.OpenAI("text-davinci-003")
def example():
# we can pre-define valid option sets
valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]
# define the prompt
program = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
"description": "{{description}}",
"name": "{{gen 'name'}}",
"age": {{gen 'age' stop=','}},
"armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
"weapon": "{{select 'weapon' options=valid_weapons}}",
"class": "{{gen 'class'}}",
"mantra": "{{gen 'mantra'}}",
"strength": {{gen 'strength' stop=','}},
"item_ids": [{{#geneach 'numbers' num_iterations=3}}
{{gen 'this'}},{{/geneach}}
]
}```""")
# execute the prompt
result = program(
description="A quick and nimble fighter.", valid_weapons=valid_weapons, llm=llm
)
print(result)
print(result.variables())
return result
if __name__ == "__main__":
example()
The outputs of the print statements are as follows:
print(result)
The following is a character profile for an RPG game in JSON format.
```json
{
"description": "A quick and nimble fighter.",
"name": "Fighter",
"age": 25,
"armor": "leather",
"weapon": "sword",
"class": "warrior",
"mantra": "Strength in numbers.",
"strength": 8,
"item_ids": [
1,
2,
3
]
},
]
}```
print(result.variables())
{
'llm': <guidance.llms._openai.OpenAI object at 0x101205df0>,
'description': 'A quick and nimble fighter.',
'valid_weapons': ['sword', 'axe', 'mace', 'spear', 'bow', 'crossbow'],
'name': 'Fighter',
'age': ' 25',
'armor': 'leather',
'weapon': 'sword',
'class': 'warrior',
'mantra': 'Strength in numbers.',
'strength': ' 8',
'numbers': [' 1', ' 2', ' 3\n ]\n}']
}
Most of the problem here seems to be that the generation of the final number in the array goes a bit wonky - not sure if I could tweak my prompt, or if it's a deeper issue.
System info
- Mac OS:
- Guidance Version 0.0.48
You're not constraining the generation to numbers only, so the LLM completes the JSON and then is told "whoops, you need to add more here." Adding a pattern to only allow a variable amount of number characters for each generation should fix it
Yes fair point, though it appears that I can't provide a pattern if I want to use an OpenAI model here so I think my specific problem persists (although happy to admit it might not be a very common case)
Hi @AutonomicPerfectionist, just curious, do you know how Guidance enforces LLMs to generate valid JSON syntax?
I haven't looked into the code but most other methods I've seen do it by biasing the output logits. If the logit is forced to negative infinity then it is essentially impossible for the LLM to "choose" the corresponding token.
A very simplistic way to dynamically bias logits according to a pattern is to loop over the tokens that each logit corresponds to and check if it results in a partial match of the regex. If not, then you set the logit to negative infinity. Similar methods can be applied to selections or other types of restricted responses.
If that's how Guidance does it, then it can force a response to fit a particular pattern through regular expressions or through selections, etc.
Another part of Guidance that helps with this is it combines the prompt with generations. So you can fill in the parts of the JSON that you know must exist and then only ask the LLM to generate the unknown parts, as shown above.
I haven't looked into the code but most other methods I've seen do it by biasing the output logits. If the logit is forced to negative infinity then it is essentially impossible for the LLM to "choose" the corresponding token.
A very simplistic way to dynamically bias logits according to a pattern is to loop over the tokens that each logit corresponds to and check if it results in a partial match of the regex. If not, then you set the logit to negative infinity. Similar methods can be applied to selections or other types of restricted responses.
If that's how Guidance does it, then it can force a response to fit a particular pattern through regular expressions or through selections, etc.
Another part of Guidance that helps with this is it combines the prompt with generations. So you can fill in the parts of the JSON that you know must exist and then only ask the LLM to generate the unknown parts, as shown above.
Thank you so much! Can people bias the output logit when using LLM APIs?
The cleanest way would be to set pattern='[0-9]+, but as you noted that doesn't work for OpenAI models.
Another easy fix is to get the model to stop generation at every comma, bracket, or newline:
import guidance
import os
llm = guidance.llms.OpenAI("text-davinci-003")
# we can pre-define valid option sets
valid_weapons = ["sword", "axe", "mace", "spear", "bow", "crossbow"]
# define the prompt
program = guidance("""The following is a character profile for an RPG game in JSON format.
```json
{
"description": "{{description}}",
"name": "{{gen 'name'}}",
"age": {{gen 'age' stop=','}},
"armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
"weapon": "{{select 'weapon' options=valid_weapons}}",
"class": "{{gen 'class'}}",
"mantra": "{{gen 'mantra'}}",
"strength": {{gen 'strength' stop=','}},
"item_ids": [{{#geneach 'numbers' num_iterations=3}}
{{gen 'this' stop=[',', ']', '\\n']}},{{/geneach}}
]
}```""")
# execute the prompt
result = program(
description="A quick and nimble fighter.", valid_weapons=valid_weapons, llm=llm
)
This gets it done:
print(result.variables()['numbers]')
[' 1', ' 2', ' 3']
