ReLU in PositionwiseFeedForward

[zhiqihuang]

Here i is the index of self.layers, therefore it is always less than the length of self.layers.

Probably you mean

if i < len(self.layers) - 1

Then no ReLU and Dropout after the last positionwise layer.