Adding support for Conv1d

[EskildAndersen]

Feature request

Can you add support for nn.Conv1d in dora? Like this issue did for LoRA: https://github.com/huggingface/peft/issues/2241.

It would be very valuable my masters thesis.

Motivation

Yes I want to use DoRA along with LoRA

Your contribution

Maybe

[githubnemo]

Hi, thanks for taking an interest in Conv1d support for DoRA.

We're currently refactoring how LoRA variations such as DoRA are implemented (https://github.com/huggingface/peft/pull/2443). Once this is done we can think about adding support for Conv1d.

Maybe you're interested in providing an implementation?

[githubnemo]

#2443 is now merged. With the new abstraction in place, it should be easier than before to add support for nn.Conv1d. Are you interested in taking this @EskildAndersen?

[EskildAndersen]

It should just be adding :

class DoraConv1dLayer(_DoraConvNdLayer): def __init__(self, fan_in_fan_out): super().__init__(fan_in_fan_out) self.conv_fn = F.conv1d

to dora.py and alter

if weight.data.ndim >= 3: lora_weight = torch.mm(lora_B.flatten(start_dim=1), lora_A.flatten(start_dim=1)) lora_weight = lora_weight.reshape(weight.shape) else: lora_weight = lora_B @ lora_A

so that >= is 3 instaed of 4, since Conv1d weights are 3D: [out_channels, in_channels, kernel_size]. Lastly in layer.py we should not raise notimplemented error but the new DoraConv1dLayer:

`class Conv1d(_ConvNd): # Lora implemented in a conv1d layer def init(self, *args, **kwargs): super().init(*args, **kwargs) if not self._kernel_dim == 3: raise ValueError(f"Conv1d layer kernel must have 3 dimensions, not {self._kernel_dim}") self.conv_fn = F.conv1d

def _get_dora_layer_class(self):
    return DoraConv1dLayer

`

[BenjaminBossan]

@EskildAndersen I think it's easiest for us to review your proposed changes if you create a draft PR and commit your changes. Could you please do that?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[githubnemo]

This should be done with https://github.com/huggingface/peft/pull/2531. Closing. Thanks!