[Feature] GPU Support for MoE models

[KyberNull]

trafficstars

Feature Request

Are there any plans of supporting MoE models in the future? Models like the Granite3.1-3B and Mixtral-8x7B are popular MoE models. Their main appeal being high inference speeds and low resource usage, making them a good choice for in device usage. Currently only CPU inference is supported and is giving promising results, further GPU support using Vulkan should drastically improve inference speeds