llama-node
llama-node copied to clipboard
What files are compatible?
I have read the document below. https://llama-node.vercel.app/docs/start
So, I looked for a model to use in the source code at the site below. https://huggingface.co/models?search=ggml
Then an error like this appears. What files should I get?
llama.cpp: loading model from model/llama-2-7b-chat.ggmlv3.q2_K.bin
error loading model: unrecognized tensor type 10
llama_init_from_file: failed to load model
node:internal/process/promises:289
triggerUncaughtException(err, true /* fromPromise */);
^
[Error: Failed to initialize LLama context from file: model/llama-2-7b-chat.ggmlv3.q2_K.bin] {
code: 'GenericFailure'
}
code:
import { LLM } from "llama-node";
import { LLamaCpp } from "llama-node/dist/llm/llama-cpp.js";
const model = "model/llama-2-7b-chat.ggmlv3.q2_K.bin";
const llama = new LLM(LLamaCpp);
const config = {
modelPath: model,
enableLogging: true,
nCtx: 1024,
seed: 0,
f16Kv: false,
logitsAll: false,
vocabOnly: false,
useMlock: false,
embedding: false,
useMmap: true,
nGpuLayers: 0
};
const template = `How are you?`;
const prompt = `A chat between a user and an assistant.
USER: ${template}
ASSISTANT:`;
const run = async () => {
await llama.load(config);
await llama.createCompletion({
nThreads: 4,
nTokPredict: 2048,
topK: 40,
topP: 0.1,
temp: 0.2,
repeatPenalty: 1,
prompt,
}, (response) => {
process.stdout.write(response.token);
});
}
run();