karpathy
Chat functionality requires big 7B model
Currently none of the tiny stories model is trained on chat, so a big llama 7B model is required to have some sensible output.
Considering that chat is assumed to be a main feature here, IMHO it seems a bit against the concept of this project that no small model can be used on it.
@karpathy thanks for your quick answer.
I have tested with tiny stories before posting, of course, and result is garbage.
Maybe due to the LLama 2 chat schema
// render user/system prompts into the Llama 2 Chat schema
if (!system_prompt.empty()) {
prompt = string("[INST] <<SYS>>\n") + system_prompt + "\n<</SYS>>\n\n" + prompt + " [/INST]";
} else {
prompt = string("[INST] ") + prompt + " [/INST]";
}
that is totally strange to tiny stories dataset.
Maybe would be good enough to just train tinystories on that format: collecting all dialogs, i.e. extracting text within quotes from generations like:
./run /tmp/ramdisk/model110m.bin -i "Lily talks to her mom and say:"
Lily talks to her mom and say: "Mom, I am hungry. Can we have some pie?"
Her mom smiles and says: "Sure, Lily. I made some pie for us. It is brown and sweet and crunchy. Do you like pie?"
Lily nods and says: "Yes, I like pie. But why is the pie brown and sweet and crunchy?"
Her mom says: "Because it has chocolate chips in it. Chocolate chips make the pie brown and sweet. Do you know what they mean?"
Lily shakes her head and says: "No, I do not know. How do you make the pie brown and sweet?"
Her mom says: "I use a special spoon to stir the chocolate chips in the pie. Then the chocolate chips go through the dough and make it brown and sweet. Do you want to try some?"
Lily says: "Yes, please. I want to try some."
Her mom gives her a small piece of the brown pie and Lily takes a bite. She smiles and says: "Mmm, this pie is good.
and make a dataset out of it. It shouldn't necessarily be question->answer like, just a dialog....but I'm no expert at all here, so just guessing...
Here another approach.
I wrote a script to extract dialogs directly from TinyStories dataset:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
def extract_dialog(story):
quotes = re.findall('"([^"]*)"', story)
if len(quotes) < 4:
return None, 0
user = True
dialog = ""
for quote in quotes:
if user:
dialog += "[INST] " + quote + " [/INST]\n"
else:
dialog += quote + "\n"
user = not user
dialog += "<|endoftext|>\n\n\n"
return dialog, len(quotes)
def main():
filename = sys.argv[1]
out_filename = sys.argv[2]
story_cnt = 0
quote_cnt = 0
story = ""
with open(filename, "r", encoding="utf-8") as file, \
open(out_filename, 'w', encoding="utf-8") as out:
for line in file:
if line != "<|endoftext|>\n":
story += line
continue
dialog, num = extract_dialog(story)
story = ""
if not dialog:
continue
out.write(dialog)
story_cnt += 1
quote_cnt += num
if (story_cnt % 1000) == 0:
print(story_cnt, quote_cnt)
print(story_cnt, quote_cnt)
if __name__ == "__main__":
main()
Using like:
./extract_dialogs.py TinyStoriesV2-GPT4-valid.txt dialogs.txt
I got something like:
[INST] Tom, can I have some blocks too? [/INST]
No, these are mine. Go find your own,
[INST] Mommy! Daddy! [/INST]
Are you okay, kids?
[INST] We're okay, Mommy. But our toys are broken, [/INST]
I'm sorry, Lily. But toys are not important. You are important. We are safe and together. That's what matters,
[INST] Lily, I'm sorry I did not share with you. You can have all the blocks you want. I love you, sister, [/INST]
<|endoftext|>
[INST] Lily and Tom, what did you do? You ruined my pot and my room. You are very naughty. You have to clean up everything. And you have to say sorry. [/INST]
Sorry, Mom. We love you. We will clean up. Please don't be mad.
[INST] I love you too, but you have to be careful. You can't touch my things without asking. And you can't make a mess like this. You have to learn to be more tidy and respectful. [/INST]
We will, Mom. We will.
<|endoftext|>
[INST] Can I help you? [/INST]
Yes, Sue. We need a new handle for the door. Can you ask dad if he has one?
[INST] Dad, do we have a new handle for the door? [/INST]
I am not sure, let's look together.
[INST] Dad, let's use a chair to stand on. [/INST]
No, Sue. That is not safe. Let's ask mom for help.
<|endoftext|>
[INST] Mummy! What is that thing? [/INST]
That, my darling, is a cauliflower. It's very yummy!
[INST] No! Mummy, don't part it! [/INST]
Oh, little one, we need to part the cauliflower so that it's ready to eat
[INST] Don't worry, sweetheart. I'll cut the cauliflower into very small pieces, and then you can play with it [/INST]
Really? Oh yay! Thank you mummy!
<|endoftext|>
etc...
@xefoci7612 fun idea! This would def work as a finetuning dataset to create a TinyStoriesChat. Of course, the chat model would be constrained to the "universe" of tiny stories, which people might find confusing in downstream use, as it doesn't have any general knowledge outside of these.
@karpathy well, it would be more a role-playing game than a chat: user impersonates a kid or mom, or a little friend and the model chats more or less coherently.
But apart from the specific content of the TinyStories universe, IMHO this nice project has a tutorial value because allows people to toy (sorry for the pun) with transformers code, even if they don't have the resources to run a 7B model on their PC (FWIW I don't)...and the new chat feature is a good one, it would be great if the people could hack on that too.