syncode icon indicating copy to clipboard operation
syncode copied to clipboard

Another case where output does not follow grammar

Open 5ila5 opened this issue 6 months ago • 0 comments

I did some more testing and found another case where the output did not match my defined grammar:

grammar = """
start: compound_statement

compound_statement: statement | (compound_statement ";" statement)

statement: def_var | use_var

def_var: DEF_VAR_NAME " = " VAR_VALUE

DEF_VAR_NAME: VAR_NAME

use_var: "use("i VAR_NAME ")"

VAR_NAME: /[a-zA-Z_][a-zA-Z0-9_]*/

VAR_VALUE: /[a-zA-Z0-9_]+/
"""

model_name = "microsoft/phi-2"
syn_llm = Syncode(model=model_name, grammar=grammar, max_new_tokens=200, mode="grammar_strict")

prompt = "repeat after me: a=1;b=2;c=3;use(a)\n"
print(syn_llm.infer(prompt, debug=True)[0])

results in the output:

a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use(a)
a = 1;b = 2;c = 3;use

So it just ignores that there needs to be a ; after the use(a) and I'm not sure why.

Is there something wrong with my grammar? or is this somehow related to #208?

5ila5 avatar Jun 12 '25 11:06 5ila5