nanoGPT
nanoGPT copied to clipboard
karpathy
Is this possible to produce text with few shot learning?
I trained a gpt model using this repo. I tried to produce text using few shot learning like the one below:
Message: Support has been terrible for 2 weeks...
Sentiment: Negative
###
Message: I love your API, it is simple and so fast!
Sentiment: Positive
###
Message: GPT-J has been released 2 months ago.
Sentiment: Neutral
###
Message: The reactivity of your team has been amazing, thanks!
Sentiment:
The result i get isn't something related. Does this repo enables that feature or is my model bad?
At the scale of nanoGPT basically the answer is no. ICL (in context learning) emerges a few B parameters down the road.
Then may i ask if i would fine tune the gpt model i trained on a prompt-answer dataset, can i get a kind of ChatGPT like model? The reason i want is to have a model in my language answering questions on some of the domains i want.
Thanks for the reply.
Hi! Try loading gpt-XL weights and fine tune to your prompt-answer dataset, It should be able to produce your desired output
@C080 Gpt2-XL is trained for English language, but i want it it for my language which is Turkish. Wouldn't that be a problem? Or will that work but won't be performing enough?
It could pick up Turkish if it has been trained on a multi-lingual dataset with turkish inside! Anyway try using two layers of Google Translates after & before so all the reasoning happens in english!
@Yusuf-YENICERI
Message: Support has been terrible for 2 weeks...
Sentiment: Negative
###
Message: I love your API, it is simple and so fast!
Sentiment: Positive
###
Message: GPT-J has been released 2 months ago.
Sentiment: Neutral
###
Message: The reactivity of your team has been amazing, thanks!
Sentiment:
This is totally possible if you scale this a lot, but there are much better models for this like bert finetuned or sentiment analysis, my repo uses a similar style, but for chat messages, like so:
<human> ... <endOfText>
<bot> ... <endOfText>
