dottxt-ai
Update logits array in-place
What behavior of the library made you think about the improvement?
The current structured generation code is creating a -inf copy of the logits array and setting the allowed token ID indices to the corresponding values in the original logits array. See here.
How would you like it to behave?
When possible, the original logits array should be updated in-place and completely avoid creating a new array. This change would likely require the set of disallowed token IDs instead of the allowed ones.
This is dead code which should be cleaned up after
https://github.com/dottxt-ai/outlines/pull/1010
This is dead code which should be cleaned up after
#1010
The relevant code now seems to be here; is that what will be updated in #1010?
No, that is a problem, thanks for pointing it out. I'll look at it once my current task is complete.
#1192 does not appear to update the logits arrays in place, at least not without creating another array the size of the logits.
Looks like this might need to be moved after https://github.com/dottxt-ai/outlines/issues/1206.