keras-team
Write a KerasNLP text generation guide
We can write one mega guide, or a few ones covering the following topics:
- Use
keras_nlp.models.GPT2CausalLMto generate text. - Finetune
keras_nlp.models.GPT2CausalLMto generate text matching a certain style, e.g., poem generation, which can be multi-language. - Text generation algorithm basics - a brief guide over popular text generation algorithms, and show how to apply them via
keras_nlp.samplers.XXX. - Train a chatbot (GPT2 may not perform well, but idk)
- Other interesting stuff
I am going to start with GPT2 finetuning, and we can discuss if we need the rest and how.
Hi @chenmoneygithub, I know you might work on the finetuning GPT on poem generation already. I found this topic really interesting to contribute. I was wondering do you need any assistance with this?
It seems that you have also had an example of keras_nlp.samplers https://github.com/keras-team/keras-nlp/blob/cd16b3d2d54c8023652861771703211063580824/keras_nlp/models/gpt2/gpt2_causal_lm.py#L68-L78
Do you need more help with that?
Specifically, I was thinking about helping with training a chatbot, if there is an opportunity to do so.
Many thanks, Junfan
@chenmoneygithub Recently I was studying more on text generation and I'd like to work on keras_nlp.models.GPT2CausalLM to generate text & writing a brief guide over popular text generation algorithms.
@xihajun Junfan, sorry for the late reply, I have been in a global trip since Feb 15th, so missed lots of threads. To your question - We are still adjusting sampler and other generation APIs to improve the performance and code quality, once they are in a stable stage, we will start welcoming contributions for those! Chatbot is definitely one interesting task! For your reference, there is a thing called DialoGPT, which is a chatbot based on GPT2. IMO a tutorial on how to finetune GPT2 to become a chatbot would have high impact, and also very fun! Please stay tuned!
@Cyber-Machine Thanks for your interest!
A guide on generation algos sounds good! Do you have a skeleton in your mind, like what algorithms do you want to cover?
For keras_nlp.models.GPT2CausalLM, we are still making a few changes. But if you find something wrong with the current API, or something you find fun to add to that API, feel free to open an issue and we can discuss from there. Thanks!
@chenmoneygithub I was thinking to make guide using GPT2, sure I will be looking through that API and share my thoughts.
Once it's finished I'll make a guide for text generation using GPT2.
Meanwhile is there some work that I can assist you?
Thank you so much for pointing out DialoGPT @chenmoneygithub
That's exactly what I'm keen to help! I will see if I can give it a go to train a chatbot and write an interesting fine-tuning 101 tutorial.
@chenmoneygithub I have been working with GPT2 recently and have been reading up a lot on it, I would love to work on guides showcasing GPT2 's text generation capabilities. We could use the persona chat dataset and train a chatbot using that. We could also showcase the personalized dialog generation as done in personaGPT. I would love to help if any is required.
