keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Use preallocated buffer for text generation

Open chenmoneygithub opened this issue 2 years ago • 0 comments

The current text generation implementation relies on tf.concat, which is not XLA compatible. Instead we can use a pre-allocated buffer to hold generated tokens.

As we are currently using generation util mainly for demo/guide, we can revisit this issue when we decide to bake this util into model.

chenmoneygithub avatar May 18 '22 17:05 chenmoneygithub