Metalhead.jl
Metalhead.jl copied to clipboard
Add more options to `ViT`
This modifies the current ViT API to add more options - notably, there is now a optional prenorm/postnorm toggle. There's also an option to make class tokens disappear completely, and also to allow class tokens to be before the positional embedding as in DeIT-III. This makes some other cleanup changes as well. The API is more congested for now but I thought I'd get this in before I start working on the other ViTs - maybe there's some potential for extracting common stuff out there.
Needs #174 to land before this makes sense. Also documentation is pending
what's status now ?
I've had less time to work on this in the recent past, but I'm going to try and push some of these refactors through in the next few months. However, there's some work that's also happening on the Attention implementations around the Flux ecosystem - I suspect any reforms to the ViT will wait on that work to land