keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Move generation functionality to base classes

Open mattdangerw opened this issue 1 year ago • 1 comments

We want it to be easy to add more generative models to the repository, and we want it to be easy to add new features related to generation.

Currently quite a bit of common functionality for generation live in every XXCausalLM subclass, which makes it hard to add new models and build new generation features. I think we can pull a lot of this functionality down into the base class. I also as part of this work we should move the compiled while loop out of the sampler class and into the CausalLM base class, which will make it much easier to write custom generative loops (e.g. would help integrating with JetStream).

mattdangerw avatar Sep 22 '24 22:09 mattdangerw