Metalhead.jl icon indicating copy to clipboard operation
Metalhead.jl copied to clipboard

Add more options to `ViT`

Open theabhirath opened this issue 3 years ago • 2 comments

This modifies the current ViT API to add more options - notably, there is now a optional prenorm/postnorm toggle. There's also an option to make class tokens disappear completely, and also to allow class tokens to be before the positional embedding as in DeIT-III. This makes some other cleanup changes as well. The API is more congested for now but I thought I'd get this in before I start working on the other ViTs - maybe there's some potential for extracting common stuff out there. Needs #174 to land before this makes sense. Also documentation is pending

theabhirath avatar Jul 24 '22 09:07 theabhirath

what's status now ?

zsz00 avatar Jan 08 '23 12:01 zsz00

I've had less time to work on this in the recent past, but I'm going to try and push some of these refactors through in the next few months. However, there's some work that's also happening on the Attention implementations around the Flux ecosystem - I suspect any reforms to the ViT will wait on that work to land

theabhirath avatar Jan 08 '23 16:01 theabhirath