mnn-llm icon indicating copy to clipboard operation
mnn-llm copied to clipboard

Published 20 hours ago • verified publisher icon wangzhaode

Support for Lora adapter layers for LLM inference

Open TejasRavichandran1995 opened this issue 1 year ago • 2 comments

the llm-export utility https://github.com/wangzhaode/llm-export seems to have support to directly export a lora.mnn file during conversion in the llm_export.py .

However , it seems to me the framework does not yet support inference with lora.mnn exported file. Any pointers regarding this would be useful :). @wangzhaode

TejasRavichandran1995 avatar Apr 19 '24 10:04 TejasRavichandran1995

MNN-2.9.0 will support apply lora on device. But now there are some accuracy problems caused by quantization.

wangzhaode avatar Apr 19 '24 10:04 wangzhaode

Sure. Thanks @wangzhaode . Any planned rough timelines on the 2.9.0 release?

TejasRavichandran1995 avatar Apr 19 '24 10:04 TejasRavichandran1995

Marking as stale. No activity in 30 days.

github-actions[bot] avatar May 20 '24 09:05 github-actions[bot]