acon icon indicating copy to clipboard operation
acon copied to clipboard

Published 20 hours ago • verified publisher icon nmaac

请问ACON能用在全连接吗，怎么用

Open onlinehuazai opened this issue 3 years ago • 1 comments

onlinehuazai avatar Jun 06 '21 04:06 onlinehuazai

可以用在全连接层，比如你的tensor shape是 (batch, width)，你可以简单修改 https://github.com/nmaac/acon/blob/main/acon.py#L13-15 为

    self.p1 = nn.Parameter(torch.randn(1, width)) 
    self.p2 = nn.Parameter(torch.randn(1, width))    
    self.beta = nn.Parameter(torch.ones(1, width))

nmaac avatar Jun 07 '21 03:06 nmaac