The output is cropped?

[rekire]

I created a test for like this:

void main() {
  test('Smoke-Test', () {
    final found = Llama.libraryPath != null && File(Llama.libraryPath!).existsSync();
    print('Using libraryPath: ${Llama.libraryPath} found: $found');
    print('Current path: ${Directory.current.path}');

    String modelPath = 'Oolel-Small-v0.1.Q4_K_S.gguf';
    Llama llama = Llama(modelPath);

    llama.setPrompt('Tell me a joke');
    var answer = '';
    while (true) {
      var (token, done) = llama.getNext();
      answer += token;
      if (done) break;
    }
    print('Joke: $answer');

    llama.dispose();
  });
}

You can download the model via:

wget https://huggingface.co/mradermacher/Oolel-Small-v0.1-GGUF/resolve/main/Oolel-Small-v0.1.Q4_K_S.gguf

If you also need the so file use this:

sudo apt-get update
sudo apt-get install build-essential libcurl4-openssl-dev
mkdir build
cd build
git checkout b4138
cmake .. \
  -DLLAMA_FATAL_WARNINGS=ON \
  -DLLAMA_CURL=ON \
  -DGGML_RPC=ON \
  -DBUILD_SHARED_LIBS=ON \
  -DLLAMA_NATIVE=ON
cmake --build . --config Release -j $(nproc)
cp src/libllama.so ../..

The output is something like:

Joke: about science or something related to that topic.

It seems that there are some words missing. What is wrong with my code? It is more or less a fork of example/test.dart.

[aesthetic-factory]

I guess you need to format your prompt into a chat template, such as chatml, before calling setPrompt. Like this.

<|im_start|>user 
Tell me a joke
<|im_end|> 
<|im_start|>assistant 

Or using a tokenizer

[netdur]

is this still an issue?

[rekire]

I added the "markup" this way:

    llama.setPrompt('''<|im_start|>user 
        Tell me a joke
        <|im_end|>
        <|im_start|>assistant''');

It still seems to be cropped (@netdur so I trend to say yes):

✅ Smoke-Test Using libraryPath: libllama.so found: false Current path: /home/runner/work/llama_cpp_dart_fork/llama_cpp_dart_fork Joke: What is the capital of France?

Ah, I must correct you, it is not a capital, but it's known for its historical landmarks. What a

🎉 1 test passed. ^Source

If I do something stupid feel free to link some documentation I really don't know it better. Regarding the tokenizer @aesthetic-factory do you have a code example?