nanoGPT
nanoGPT copied to clipboard
karpathy
Making nano chatgpt
Even though it expands beyond the scope of nanoGPT, a nanoChatGPT also came to my mind: replace web-site/blog search engine with nanoChatGPT answering questions based on information of a limited yet strong focused textset or factset.
I suggest we keep this thread/issue open so people can comment with links of "nano" ChatGPT like projects.
chatgpt training process is publicly available, but the results are highly dependent on the fine-tuned datasets that openai, which is hard to do
I think we can try webgpt, which is a more engineered solution
https://arxiv.org/abs/2112.09332 https://openai.com/blog/webgpt/
of course these are far beyond the scope of nanoGPT/MakeMore courses but, exercises after class are really fun
Another effort at https://open-assistent.io, code at https://github.com/LAION-AI/Open-Assistant
I saw a project about chatgpt and felt compelled to share https://github.com/hpcaitech/ColossalAI
They have made a very magical optimization of the training process. It is said that chatgpt(small) can be fine-tuned on a single card, and there is a complete training process code
I've started building a nanoChatGPT as a fork of Karpathy's chatGPT. I also introduce a new idea for training by backpropagating through the reward function using the Gumbel-Softmax trick rather than policy gradient (PPO).
It works for a basic example but is still very crude though and far from being useful at scale. Sharing here in case anyone wants to try:
https://github.com/sanjeevanahilan/nanoChatGPT
We have released nanoT5 for pre-training and evaluating T5-style (Encoder-Decoder) models. You can use it to pre-train your own model in one day on a single GPU :).
