Default expansion factor of 2 for inner dimension in conv module not present in other implementations

[robflynnyh]

Hey lucidrains, firstly thanks for providing this implementation!

I've noticed in the ConformerConvModule the inner dimension is increased by an expansion_factor which defaults to 2 inner_dim = dim * expansion_factor In the implementations by espnet (https://espnet.github.io/espnet/_modules/espnet/nets/pytorch_backend/conformer/convolution.html) and nvidia (https://github.com/NVIDIA/NeMo/blob/94a464fc4eb2927140940cc835a0ab69ee0347b5/nemo/collections/asr/parts/submodules/conformer_modules.py) the dimensions of the convolutional module are the same as the encoder dimensions i.e no expansion.

Obviously this just defaults to two and is configurable but - I'm wondering whether this is unintentionally the default, or if you think/have found this to be a good use of parameters?

the conformer paper is a bit sparse on details but I think the expansion factor they refer to in there corresponds to the out_channels for the pointwise convolution, which is also applied here i.e nn.Conv1d(dim, inner_dim * 2, 1),

Thanks ! (: