Rémi Louf
Rémi Louf
How about looping over the entire vocabulary and store the tokens that accept `mor` as a prefix. Then, in the unconstrained case the first state of the FSM would have...
Yes I think that's the right approach. There's some stuff to figure out in terms of design, but otherwise looks good.
We're getting really close. There are a few design changes remaining, and mostly we should have comprehensive tests before merging.
I rebased your branch on `main` after a big refactor of the `FSM` interface. I will take a closer look this week.
Yes! I'm currently thinking about how we could integrate that to the logits processors since most integration are going to use this :)
I think it is time to revisit this as #966 is about to be merged and the custom sampling loop will be removed. We can still implement this via passing...
Could you rebase on `main` now that #966 was merged?
It's hard to evaluate the performance improvement ahead of time; at this point there is no other choice but to implement and benchmark. Afaiu this is close to what was...
Said PR was making several changes at the same time, with different impacts on the compile time, which made it difficult to evaluate the performance impact. We asked for the...