Improve CFG Correctness

[lapp0]

Per https://github.com/dottxt-ai/outlines/pull/1067#issuecomment-2251396332

Here are some necessary performance and correctness improvements.

Ensure parser allows ambiguous terminals

The grammar ?start: /ab*/ /bc?/ with a generation of abbbb doesn't allow a next token of c, it requires bc.

Allow skipping rule

The grammar ?start: "a" ("b" | "c")* "d" with a generation of a doesn't allow a next token of d

Incorrectly Over-Constrained

• arithmetic_lots_of_ops.arithmetic.test - guide doesn't allow generation of eos token at end