[LoRA] Adds support for bias in LoRA

[followumesh]

Motivation PEFT, https://github.com/foundation-model-stack/fms-hf-tuning includes support for tuning LoRA bias. This PR enables bias for lora, so the adapters with bias will work with vLLM.

Changes Included

• LoRA bias support for different types of modules.
• LoRA bias support for fully sharded LoRA.
• Test file test-lora-bias.py

[followumesh]

@Yard1 Thanks for reviewing the PR. I have added the enable_lora_bias flag (default set to false), which prevents the allocation of lora bias tensors when false.

[njhill]

Related: https://github.com/vllm-project/vllm/pull/5930

[DarkLight1337]

To speed up the CI queue for #5905, I've cancelled the distributed tests for the latest CI run in this PR since they won't pass anyway until #5905 has been merged. Please merge main into your branch after that happens so that the CI can pass once again.

[followumesh]

@Yard1 Thanks for reviewing. I've added an e2e test for the lora_bias support.

[njhill]

@followumesh you need to run ./format.sh to fix the linting errors

[njhill]

@followumesh apologies, this needs another conflict resolution!

[followumesh]

@Yard1 @njhill I have included the e2e test and merged the recent changes. Can you please review the commit? Thanks

[followumesh]

@njhill I have addressed your comments above. Can you please review this again? Thanks

[njhill]

@followumesh there are a few failures in the existing LoRA tests which look related.

[followumesh]

@njhill All LoRA tests are sucessful now.

[jeejeelee]

Thanks for complete this feature. I have two question about this featue:

• Is this feature compatible with PEFT?
• Have you done any benchmarking? Adding --enable-lora-bias seems to inevitably impact performance.