[Feature] Use xgrammar as default grammar backend to aviod I/O errors while using Outlines in a multi-node setting

[shuaills]

Checklist

• [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• [x] 2. Please use English, otherwise it will be closed.

Motivation

related issues: #3375 related discussiton: #vllm 4193 related pr: https://github.com/sgl-project/sglang/pull/3379

Related resources

xGrammar stores its cache in RAM instead of disk, avoiding file system conflicts. Cache size is small (typically <0.5MB per schema), meaning it doesn't require persistent disk storage. xGrammar is thread-safe, ensuring it can run across multiple Slurm nodes without concurrency issues.

[adi-kmt]

Hey @shuaills, can i pick this up?

[liusy58]

I'd like to work on this, too~

[rlouf]

You can deactivate Outlines' cache using:

from outlines import caching
cachine.disable_cache()

And use functool's lru_cache instead.

[vincentzed]

See #6601 since we updated to the default to be xgrammar alreaadya in the meantime, but one docs mention doesn't use it