Chunk adapter for fused qkv_proj

[yxli2123]

trafficstars

As microsoft/Phi-3-mini-4k-instruct is using fused qkv_proj and up_gate_proj, LoRA adapters should be applied to each chunk instead of the entire fused matrix.

The change lies in (1) adapter initialization and (2) forward computing.

For (1), we create n_chunk adapters for one fused matrix, default-chunk-0, default-chunk-1, ... for both lora_A and lora_B.

For (2), we iterate the adapter by adapter_name. Once we find its chunk, we only update the associated dense result.

We have run the make style and make quality.

[BenjaminBossan]

Thanks for providing this PR. Indeed, as of now, we don't have any special provision for LoRA to deal with fused QKV layers (or any fused layers). For some models that used fused QKV, we just apply the LoRA adapter on the whole fused layer. I haven't thought this through completely, but I wonder if this should not be good enough. Do we really need to chunk the LoRA adapter?

Where there is definitely a problem is if we have fused QKV but we want to apply LoRA only to Q and V, for instance. This is currently not possible, but AFAICT, your PR wouldn't change that situation, is that right?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.