How to make sentences make more sense?

[gitihobo]

Is it the amount of iterations? How do I add sense and variety to my llm?

[JKHenry520]

the same confusion。add data ? add layers? what's the smallest layer count？

[gitihobo]

Hope someone helps us soon

[the-crypt-keeper]

how much sense do you expect? there's some ideas in the TinyStories paper: https://arxiv.org/abs/2305.07759

their dataset is here: https://huggingface.co/datasets/roneneldan/TinyStories

i have used it to pre-train and it's definitely improved the models (at expense of much compute time..)

[gitihobo]

and what are we supposed to do to in terms of settings to archive similar results?

[Majdoddin]

This bug affects the quality negatively: https://github.com/karpathy/nanoGPT/issues/320

[VatsaDev]

GPT-2 is glorified auto complete with the ability to make sentences, If you want better sentences, fine tune it. I have personally had pretty good success with finetuning gpt-2-medium into making conversation, sentences, and even small paragraphs.

[gitihobo]

So how do you finetune it?

[VatsaDev]

theres the Finetuning section In the readme, read that, but the command is $ python train.py config/finetune_shakespeare.py

[gitihobo]

Thank you, know I do know how to fine tune, what I am not sure about is the data, how do I get the amount of text necessary and how do I have to format it to make a good fine tune?

[VatsaDev]

I have addressed many of these issues in my repo NanoChatGPT, all the details are in the README. I formatted my data like this

<human> ... <endOfText>
<Bot> ... <endOfText>
<human> ... <endOfText>
<Bot> ... <endOfText>
<human> ... <endOfText>
<Bot> ... <endOfText>

since my data was conversational, I took conversation corpuses, the whole list is on my repo readme, but one dataset I found to be pretty great was the personachat dataset