tf-nlp-blocks

tf-nlp-blocks copied to clipboard

Unneeded ReLU in Transformer-match

2

Hi, Thank you for sharing this great repository! I think that the ReLU in the k/q/v projections is unneeded, or at least, inconsistent with the Transformer paper: [https://github.com/hanxiao/tf-nlp-blocks/blob/master/nlp/match_blocks.py?fbclid=IwAR0HdDwaSpPZbhwi6BfzWY0P0ZIMOWAobZU5aPtvpZO2d3MTOEXrwkQzY5A#L113 ](https://github.com/hanxiao/tf-nlp-blocks/blob/master/nlp/match_blocks.py?fbclid=IwAR0HdDwaSpPZbhwi6BfzWY0P0ZIMOWAobZU5aPtvpZO2d3MTOEXrwkQzY5A#L113 )...

urialon

1. if use_spatial_dropout == True, apply spatial_dropout to out2. 2. use max_seq_len to create lookup_table instead of seq_length.

libertatis