Benjamin Bossan

Add PyTorch 2.5.0 to CI

1

* Drop 2.1.2, update 2.4.0 to 2.4.1 * Drop Python 3.8, add Python 3.12 * Filter new torch warning about `weights_only` as that trips up some warning tests * Unpin...

[WIP][LoRA] Implement hot-swapping of LoRA

30

This PR adds the possibility to hot-swap LoRA adapters. It is WIP. ## Description As of now, users can already load multiple LoRA adapters. They can offload existing adapters or...

### 🐛 Describe the bug When running `F.scaled_dot_product_attention` with an input matrix that contains NaNs on CPU, with PyTorch 2.4, the output is a NaN matrix, but with PyTorch 2.5,...

Call for contribution: KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models

9

### Feature request There is an [existing implementation of KaSA](https://github.com/juyongjiang/KaSA) that can be used as reference for the PEFT implementation. AFAICT, the changes mainly come down to [this new parameter](https://github.com/juyongjiang/KaSA/blob/f85e88c22d0fa4cb8ab2923d7c2bf1bbec152da3/peft/src/peft/tuners/lora/layer.py#L130)....

Method comparison: Support more options for the optimizer

1

Allow setting a different optimizer, including PEFT specific ones like LoRA+. This PR requires #2468.

FIX: Use correct argument name in MultiheadAttention forward

1

The arguments of the `forward` method of `MultiheadAttention` are called `query` etc. PEFT used `x`. Therefore, if a caller uses keywords only, the argument is not assigned, resulting in an...

TST: Refactor remaining common tests to use pytest

1

Follow up to #2462 and #2478 This deals with all the remaining tests that rely/relied on `PeftCommonTester`. For each test file, I created a separate commit, which may help reviewing....

FIX Prompt learning issue with 4d attention mask

19

Resolves #2452 Some causal language models in transformers have 4d attention masks at the input preparation stage. So far, we have assumed 2d attention masks, which results in an error...

ENH: Orthogonal LoRA layer initialization (2)

2

Continuation of, and supersedes #2389 Check discussion there for further info.

Right now, the MacOS GitHub runners use `macos-13`, which uses Intel x86. This has the disadvantage that the latest torch wheels still provided for this architecture are torch v2.2.2. Later...