Tiny llama is outputting gibberish on the google colab

[DrewThomasson]

When testing it out in the google colab it's outputting gibberish,

Are there more things I have to specify like the formatting of the dataset?

I thought the google colab already did that for me by default?

[DrewThomasson]

nevrmind lol

[danielhanchen]

@DrewThomasson Oh what's the inference commands you used - what max_seq_length was set and was the model finetuned? I was planning to add a flag to check if a model is finetuned, and warn people on using too long max_seq_lengths which cause gibberish

[DrewThomasson]

@DrewThomasson Oh what's the inference commands you used - what max_seq_length was set and was the model finetuned? I was planning to add a flag to check if a model is finetuned, and warn people on using too long max_seq_lengths which cause gibberish

OK yes the default was 4096 im changing it to 2048 to see if that works

Also after I download the fine tuned model as a gguf in q4_k_m format whatnot, I tried setting it up in Ollama with these default model file settings for tinyllama All I do is change the directory

FROM /Users/admin/Documents/Test-unsloth.Q4_K_M.gguf TEMPLATE """<|system|> {{ .System }} <|user|> {{ .Prompt }} <|assistant|> """ SYSTEM """You are a helpful AI assistant.""" PARAMETER stop "<|system|>" PARAMETER stop "<|user|>" PARAMETER stop "<|assistant|>" PARAMETER stop ""

Right now then creating that in Ollama its outputting gibberish like this

sup bitch

|s

|- 1 | | |-2| | |3

Send a message (/? for help)

Ill come back and tell you if the shortening the max length works tho!

[DrewThomasson]

ok yeah looks like that fixed now its outputting like this

hey what is up? thank you for coming to the office, and i hope your day was good today. please tell me if there are any problems and i will help you. i have some new tools for you.

<|new tool|> hello, my name is

Send a message (/? for help)