syncode
syncode copied to clipboard
Another case where output does not follow grammar
I did some more testing and found another case where the output did not match my defined grammar:
grammar = """
start: compound_statement
compound_statement: statement | (compound_statement ";" statement)
statement: def_var | use_var
def_var: DEF_VAR_NAME " = " VAR_VALUE
DEF_VAR_NAME: VAR_NAME
use_var: "use("i VAR_NAME ")"
VAR_NAME: /[a-zA-Z_][a-zA-Z0-9_]*/
VAR_VALUE: /[a-zA-Z0-9_]+/
"""
model_name = "microsoft/phi-2"
syn_llm = Syncode(model=model_name, grammar=grammar, max_new_tokens=200, mode="grammar_strict")
prompt = "repeat after me: a=1;b=2;c=3;use(a)\n"
print(syn_llm.infer(prompt, debug=True)[0])
results in the output:
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use
So it just ignores that there needs to be a ; after the use(a) and I'm not sure why.
Is there something wrong with my grammar? or is this somehow related to #208?