ialacol icon indicating copy to clipboard operation
ialacol copied to clipboard

feat: rewritten to Rust/WASM with wasm-edge + wasi-nn-ggml

Open chenhunghan opened this issue 4 months ago • 0 comments

ctransformers has no updates for months, and Rust/wasm seems to be a promising candidate to make ialacol tiny and efficient, I have began the work to covert ialacol from python/FastAPI to Rust.

Candidates for backend

  • https://github.com/rustformers/llm (no updates)
  • https://github.com/huggingface/candle (is relatively slow compared to wasm-edge)
  • https://github.com/second-state/LlamaEdge (wasm-edge + wasi-nn-ggml plugin)

However, likely we will drop supports for GPTQ, and focus on GGML/GGUF format LLMs.

chenhunghan avatar Feb 04 '24 15:02 chenhunghan