Metalhead.jl
Metalhead.jl copied to clipboard
FluxML
Swin Transformer related utility function
add a function which returns a matrix of relative position index from Swin Transformer
Thank you for this contribution! Unfortunately, we don't accept PRs for isolated utilities like this. If this function is useful for constructing a larger model, then we would need to review the utility usage in the context of the complete model code (or some draft of it) to effectively evaluate the implementation. I encourage you to try implementing the full model as well!
@XWHtorrentx it is still not clear what you are trying to accomplish with this PR. As @darsnack said, we don't want to provide utility functions in this repo, but just model constructors. Are you going to implement the Swin Transformer or what?
I am currently working on Swin Transformer. It is almost done and I am testing it. Since I am new here I don’t know that I should pull a model as a whole.
2022年3月24日 15:50，Carlo Lucibello @.***> 写道：
@XWHtorrentx https://github.com/XWHtorrentx it is still not clear what you are trying to accomplish with this PR. As @darsnack https://github.com/darsnack said, we don't want to provide utility functions in this repo, but just model constructors. Are you going to implement the Swin Transformer or what?
— Reply to this email directly, view it on GitHub https://github.com/FluxML/Metalhead.jl/pull/134#issuecomment-1077333905, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXOCCC262AU3DLDFDG32LVLVBQNFJANCNFSM5QXUSZPQ. You are receiving this because you were mentioned.
I would say open a new PR for the whole model. You can target it against this branch instead of master if that would make your life easier too.
Also, we'll help guide your PR so that it is ready to merge. You don't need to worry about it being partially incomplete.
The other thing is the use of NeuralAttentionlib, which last I checked slowed the model down a lot on the CPU. Not sure if there's been any changes there - I will run some benchmarks to check
