llama.cpp On the edge llama?

On the edge llama?

Open NoNamedCat opened this issue 1 year ago • 1 comments

Sorry to ask this... But is possible to get llama.cpp working on things like edge TPU?

https://coral.ai/products/accelerator-module/

Apr 19 '23 01:04 NoNamedCat

That thing doesn't really have the memory needed for a LLM (8MB on-chip). You would have to transfer data in and out for processing and that would become the bottleneck.

Apr 19 '23 20:04 ghost

llama.cpp llama.cpp copied to clipboard

On the edge llama?

llama.cpp
llama.cpp copied to clipboard