yoyodyne
yoyodyne copied to clipboard
Nested tensor warning
The transformer logs this on creation:
UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
There are two things we can do about it:
- To silence the error message but nothing else, we can add
enable_nested_tensor=False
here. - Or we can set
norm_first=False
here.
I don't know what the consequences of the latter are, really (I have no intuitions about when to norm) so I piloted it and it doesn't seem to affect performance much.