outlines icon indicating copy to clipboard operation
outlines copied to clipboard

Add ensure_ascii parameter to json function

Open lm-cyber opened this issue 8 months ago • 3 comments

Add ensure_ascii parameter to json function in generate

lm-cyber avatar Apr 19 '25 19:04 lm-cyber

Thank you for contributing! What was the issue prior to this PR?

rlouf avatar Apr 30 '25 20:04 rlouf

Thank you for contributing! What was the issue prior to this PR?

Hi, different models have different tokenizers and samples for learners, so for languages whose characters are not in ascii, this parameter is required, it's just that the models were not always trained on \uXXXX

Sorry for not responding right away, there was no notification.

lm-cyber avatar May 09 '25 15:05 lm-cyber

Hi @lm-cyber! The library has changed a lot recently since we released Outlines v1. I believe it makes handling the subject of your PR either though! I think what we could do is to add an optional parameter for ensure_ascii to the __init__ of the JsonSchema object in the outlines/types/dsl.py file.

RobinPicard avatar Jun 19 '25 16:06 RobinPicard

Closing it as the issue has been treated by #1672

RobinPicard avatar Jul 02 '25 04:07 RobinPicard