Trans2D

Trans2D copied to clipboard

question about code

1

in your paper ,here should be scaled dot product while element-wise multiplication in your code ``` full_attn = (query_layer.unsqueeze(-1).unsqueeze(-1) * key_layer.unsqueeze(2).unsqueeze(2).transpose(4, 6)).sum(4) time_attn = (query_layer.unsqueeze(3) * key_layer.unsqueeze(2)).sum([4, 5]).unsqueeze(-2).unsqueeze(-2) feature_attn =...

bb67ao