No LayerNorm Implementation

[Karn3003]

Hi, Thanks for the awesome implementation. It would be great if you could add Layer Norm implementation as well.

[soumickmj]

Thanks @Karn3003 Happy to hear that you found it useful. I will add Layer Norm ASAP. If you have any specific deadline, let me know.

[Karn3003]

Yeah, I am using your codebase for one of my projects and the deadline is next month. Was testing some architectures and needed layer norm for that so would be helpful if you could have implementation for that as well. Also do add following activation as well.

class ComplexGELU(nn.Module):
    def __init__(self):
        super(ComplexGELU, self).__init__()

    def forward(self, input):
        real_gelu = F.gelu(input.real)
        imag_gelu = F.gelu(input.imag)

        return torch.complex(real_gelu, imag_gelu)