socket icon indicating copy to clipboard operation
socket copied to clipboard

Support Llama.cpp as a core module

Open heapwolf opened this issue 1 year ago • 1 comments

This PR will add JavaScript bindings for llama.cpp

import { Model, Context, Grammar } from 'socket:llm'

// include a model file in your project
const model = new Model('./mistral-7b-openorca.Q4_0.gguf')

// create a grammar
const grammar = new Grammar('...') // supports (json)gbnf

// create a context
const context = new Context({ model, grammar })

// use the context to evaluate the input and produce output
context.eval(tokens, params)

heapwolf avatar Feb 03 '24 18:02 heapwolf

Google has recently released gemma models and https://github.com/google/gemma.cpp.

ansarizafar avatar Feb 24 '24 05:02 ansarizafar

added to #next branch

heapwolf avatar Jun 06 '24 06:06 heapwolf