`STOPS_AT(X, "\n")` broken when using `len(TOKENS(X)) > 100`

[freckletonj]

trafficstars

With this query, the output can include many newlines (Using Falcon 7B):

@lmql.query()
def user():
    '''lmql
    '[USER]' where STOPS_AT(USER, "\n") and len(TOKENS(USER)) > 100
    '''

I would expect it to rule out \n tokens until the length is > 100.

I saw this error: https://github.com/eth-sri/lmql/issues/144

But since they attribute it to a Llama Tokenizer bug, it's irrelevant to this issue since I'm using Falcon.

[lbeurerkellner]

Thanks for raising this. The constraint you specified according to LMQL semantics means that your variable value will be at least 100 tokens long, and, once this limit is reached, stop on the next newline. However, before then, the newline character is indeed allowed, it will just not end generation. In this sense, LMQL constraints always describe the final result, not a temporal property.

I would expect it to rule out \n tokens until the length is > 100.

This is an interesting temporal property that is currently not expressible in pure constraints, I think. An alternative, you can split it into two constraints and variables:

"[PT1]" where len(TOKENS(PT1)) > 100 and not "\n" in PT1
"[PT2]" where STOPS_AT(PT2, "\n")

The upcoming 0.7 release even brings nested queries, which would allow you to encapsulate this into a separate (constraint) function, usable in other parts of your code. See the docs on this here.

Let me know, if this make sense :)

[freckletonj]

Thanks for the fast response!

I guess I'm confused at the docs then:

https://docs.lmql.ai/en/stable/language/constraints.html

To do so, LMQL supports the STOPS_AT constraint. It takes two arguments, the first is the name of the variable to which the model output is assigned, the second is the stopping phrase. In the example below we use it to ensure that as soon as the model predicts the newline character \n the decoding of the variable THING is stopped

But if I understand you correctly, STOPS_AT doesn't mean it stops at the first occurence, it means that by the time it does stop, it's guaranteed to stop on your token of choice.

(OT, does STOPS_AT support regex?)