open_lm icon indicating copy to clipboard operation
open_lm copied to clipboard

[WIP] Attention across documents.

Open GeorgiosSmyrnis opened this issue 1 year ago • 0 comments

This adds a flag that stops attention from going across documents, identified by the EOT token.

The loss for the token right after the EOT token is ignored.

TODO: add some tests for the shape of the mask.

GeorgiosSmyrnis avatar Jan 31 '24 21:01 GeorgiosSmyrnis