Stas Bekman

@zeyugao, so I suppose https://github.com/huggingface/transformers/pull/31525 will have to be revisited as well once https://github.com/huggingface/transformers/pull/32305 has been merged.

This is exciting, thank you for finding time to working on this important need, Sourab! 1) I think this one is trivial - stash the engine into the model once...

If I manipulate the schema to swap the order of `age` vs `description` so that it ends with an integer value which is a very short string: ``` -schema =...

another thought - the model has no clue of how many tokens it can use to build the output other than based on training experience, so if it was taught...

That's a great idea, Nick! So I tried a simple logit processor that promotes the select few tokens to the top towards the end of the context window and it...

ok, this seems to be quite automated: ``` # user input max_tokens = 25 end_chars = list('"}') end_token_ids = [tokenizer.encode(x)[1] for x in end_chars] start_ending = max_tokens - len(end_chars) -...

hmm, running bigger batches, it's still failing at time, so it's not foolproof. e.g.: ``` {"age":22,"description": "This person has a strong alove they share with their partner } } }...

OK, so I switched to making my own guided generation for the last few tokens, where I prescribe the one exact character to choose: ``` end_chars = list('"}') + [tokenizer.eos_token]...

> > `{"age":22,"description": "This person has a strong alove they share with their partner } } }` > > I'm a bit surprised that this was generated since it's not...

The main problem with this solution is that the generation is still chopped off wrt to the contents of the strings - it'd be nice to be able to signal...