BoilerPlate

BoilerPlate copied to clipboard

ReLU after residual branch missing?

1

In the paper you can see the diagram of Fixup which applies the residual branch, and adds the output to the original input. Finally it applies a ReLU to this...

f90

Question about .layer_index in init_weights

2

Hi, In fixup.py: I noticed when I make the following change: ```print(b.layer_index, math.sqrt(2. / n), self.layer_index **(-0.5))``` to ```print(b.layer_index, self.layer_index, math.sqrt(2. / n), self.layer_index **(-0.5))``` That `b.layer_index` and `self.layer_index` differ......

rh314