aitextgen

aitextgen copied to clipboard

A stepwise implementation of generation (instead of using generate() wholesale) is necessary for:

• Generating text beyond max_length of the model (infinite generation via sliding windows
• Returning partial generated text to generate in "real time".

Should be similar to the Huggingface implementation using the past parameter.

Yes! I think this is what I was trying to describe on Patreon.

This implementation will be conditional on https://github.com/huggingface/transformers/pull/4581 (good timing!)

The patch has been merged on huggingface! Is this feature planned in the near future?