Wide NPU support (AMD)

[Danz17]

Is your feature request related to a problem? Please describe.

Since the market of consumer's adapting NPU backed pcs from AMD such Ryzen AI series, and localai target is the general public, it would be great to support this from the start. Describe the solution you'd like

Configuration to detect NPU, configuration to deploy NPU modules from huggingface Onix , Configuration for Hybrid support (NPU+GPU) Describe alternatives you've considered

This is core requirement, I can't think of alternative, skipping untapped resource is a waste. Additional context

[tommerty]

I too have a HX370 and would love native support for the NPU. I believe we're waiting on a rocm update from AMD to be able to actually utilize it if I'm correct? I'm only new to this realm so I might be completely off base.

[CleverLittleMaker]

+1

Would love to see this as well. :)

[CleverLittleMaker]

I too have a HX370 and would love native support for the NPU. I believe we're waiting on a rocm update from AMD to be able to actually utilize it if I'm correct? I'm only new to this realm so I might be completely off base.

I am also not an expert, but from what I understand, AMD NPU support is in Linux kernel 6.14 and later (if you are using Linux). ROCm support is not required to utilize the NPU via the AMDXDNA kernel driver. It's a matter of utilizing the proper stack, and probably needs to be done in the backend (i.e. llama.cpp or similar) I'm guessing.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.