[Feature Request] Support `torch.nn.Conv1d` layers for LoRA

[dg845]

Feature request

I would like torch.nn.Conv1d layers to be supported by LoRA. Currently, if I try to include a torch.nn.Conv1d layer in LoraConfig.target_modules, I get the following error:

ValueError: Target module Conv1d(...) is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv2d`, `transformers.pytorch_utils.Conv1D`.

I see that transformers.pytorch_utils.Conv1D is supported but it would be nice to be able to use non-transformers models with LoRA as well. [Edit: also, transformers.pytorch_utils.Conv1D is not as feature-ful as torch.nn.Conv1d (for example, it doesn't support different kernel_sizes) because it's intended to essentially be a linear layer with weights transposed for following the OpenAI GPT-2 implementation.]

Motivation

Audio models often use torch.nn.Conv1d layers so it would be easier to use LoRA with audio models if it is supported.

Your contribution

I would be willing to open a PR for this, not sure if it can supported in the same way torch.nn.Conv2d is supported or if there are any specific difficulties with torch.nn.Conv1d.

[BenjaminBossan]

Yes, thanks, it would be good to have nn.Conv1d support for LoRA. Feel free to open a draft PR early and we can give quick feedback. Take a look for instance at #1324, which also adds a new module type, for inspiration.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.