llama-node icon indicating copy to clipboard operation
llama-node copied to clipboard

What files are compatible?

Open bedcoding opened this issue 11 months ago • 0 comments

I have read the document below. https://llama-node.vercel.app/docs/start

So, I looked for a model to use in the source code at the site below. https://huggingface.co/models?search=ggml

Then an error like this appears. What files should I get?

llama.cpp: loading model from model/llama-2-7b-chat.ggmlv3.q2_K.bin
error loading model: unrecognized tensor type 10

llama_init_from_file: failed to load model
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: Failed to initialize LLama context from file: model/llama-2-7b-chat.ggmlv3.q2_K.bin] {
  code: 'GenericFailure'
}

code:

import { LLM } from "llama-node";
import { LLamaCpp } from "llama-node/dist/llm/llama-cpp.js";

const model = "model/llama-2-7b-chat.ggmlv3.q2_K.bin";
const llama = new LLM(LLamaCpp);
const config = {
    modelPath: model,
    enableLogging: true,
    nCtx: 1024,
    seed: 0,
    f16Kv: false,
    logitsAll: false,
    vocabOnly: false,
    useMlock: false,
    embedding: false,
    useMmap: true,
    nGpuLayers: 0
};

const template = `How are you?`;
const prompt = `A chat between a user and an assistant.
USER: ${template}
ASSISTANT:`;

const run = async () => {
  await llama.load(config);

  await llama.createCompletion({
      nThreads: 4,
      nTokPredict: 2048,
      topK: 40,
      topP: 0.1,
      temp: 0.2,
      repeatPenalty: 1,
      prompt,
  }, (response) => {
      process.stdout.write(response.token);
  });
}

run();

bedcoding avatar Mar 23 '24 13:03 bedcoding