yoyodyne icon indicating copy to clipboard operation
yoyodyne copied to clipboard

Nested tensor warning

Open kylebgorman opened this issue 7 months ago • 6 comments

The transformer logs this on creation:

UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True

There are two things we can do about it:

  1. To silence the error message but nothing else, we can add enable_nested_tensor=False here.
  2. Or we can set norm_first=False here.

I don't know what the consequences of the latter are, really (I have no intuitions about when to norm) so I piloted it and it doesn't seem to affect performance much.

kylebgorman avatar Jul 23 '24 12:07 kylebgorman