ialacol
ialacol copied to clipboard
feat: rewritten to Rust/WASM with wasm-edge + wasi-nn-ggml
ctransformers has no updates for months, and Rust/wasm seems to be a promising candidate to make ialacol
tiny and efficient, I have began the work to covert ialacol
from python/FastAPI to Rust.
Candidates for backend
- https://github.com/rustformers/llm (no updates)
- https://github.com/huggingface/candle (is relatively slow compared to wasm-edge)
- https://github.com/second-state/LlamaEdge (wasm-edge + wasi-nn-ggml plugin)
However, likely we will drop supports for GPTQ, and focus on GGML/GGUF format LLMs.