executorch
executorch copied to clipboard
pytorch
Does llama2 example on Android utilize HTP?
https://github.com/pytorch/executorch/blob/main/examples/models/llama2/README.md#performance
AFAIK, Qualcomm Socs have HTP to boost AI performance. Are there performance numbers using HTP or just GPU?
We can run smaller version of Llama structure in HTP, and WIP for the 7B and 8B models in HTP. cc @cccclai. Running Llama in mobile GPU is also WIP cc @SS-JIA
We can run smaller version of Llama structure in HTP, and WIP for the 7B and 8B models in HTP. cc @cccclai. Running Llama in mobile GPU is also WIP cc @SS-JIA
I'm a little confused. If HTP & GPU are both not fully implemented now, what's executorch using, just CPU?
The perf number shared in https://github.com/pytorch/executorch/blob/main/examples/models/llama2/README.md#performance is purely CPU, using XNNPACK backend.
For lowering to QCOM HTP, it's still ongoing and we only have the enablement for the small stories models right now. We're actively working on enable llama2 and improve performance number.
Well, thanks for your reply. Actually it makes sense since Qualcomm themselves do not have an open approach to run LLM on htp.
We can run smaller version of Llama structure in HTP, and WIP for the 7B and 8B models in HTP. cc @cccclai. Running Llama in mobile GPU is also WIP cc @SS-JIA
I'm a little confused. If HTP & GPU are both not fully implemented now, what's executorch using, just CPU?
You can just follow the steps to execute the llama2 on HTP. But the whole graph of llama2 is not running on HTP, running on CPU partly if some Ops/subgraph is not supported on HTP.
@CHNtentes Are you looking for running llama2 model via HTP or GPU?
It would be better if I can run llama on htp. I suppose the preformance and power consumption is better than cpu/gpu.
In case this is of interest, we provide an example for deploying TinyLlaMA-1.1B-Chat on HTP (SM8650): https://github.com/saic-fi/MobileQuant/tree/main/capp. However, the solution is pretty ad-hoc compared to executorch.
