Fix IndexError caused by invalid token IDs in CFGGuide

[RohitRathore1]

It fixes issue #1232

These changes fix the IndexError caused by invalid token IDs in allowed_tokens_concat by handling eos_token_id appropriately and adjusting token handling in CFGGuide. The updates maintain backward compatibility and ensure that existing functionality continues to work as expected.

Tested on CPU:

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:08<00:00,  1.76s/it]
Saturn
\{[ ]?"caption"[ ]?:[ ]?"([^"\\\x00-\x1F\x7F-\x9F]|\\["\\])*"[ ]?\}
{"caption":"Command module pilot Buzz Aldrin walks across the lunar surface behind the deployed Lunar folloteneer's Ramp. The bottom of a Life Science Branch leg lock is framed in a footprint on the lunar surface behind the left leg of Aldrin's suit. The videocamera on the fullmomteiner's chest is visible atop the open hatch. Apollo 11, Aug. #42; CC AS11-40-5924,"}

[rlouf]

Thank you for contributing! Could you add a test that fails on current main and passes here?

[RohitRathore1]

Thank you for contributing! Could you add a test that fails on current main and passes here?

Thank you for contributing! Could you add a test that fails on current main and passes here?

Hi @rlouf to me it doesn't seem related to my changes. Let me have a deeper look

[rlouf]

Thank you for making the change :) Is it good for review?

[RohitRathore1]

Thank you for making the change :) Is it good for review?

Hi @rlouf Yes, it is ready to review now. I have provided the correct model name which was the reason to fail oneof our test case.

[rlouf]

We deprecated the old CFG backend in favor of integrating xgrammar and llguidance.