ThunderKittens icon indicating copy to clipboard operation
ThunderKittens copied to clipboard

Published 20 hours ago • verified publisher icon HazyResearch

Add support for head dimension 128

Open perkfly opened this issue 1 year ago • 4 comments

Most recent models use hdim=128, it would be great to see that ThunderKittens also support that.

https://github.com/HazyResearch/ThunderKittens/blob/a562ed2569c45b0ffea844688594158cb7c6e858/examples/attn/h100/h100_train_atn.py#L25-L26

perkfly avatar May 16 '24 01:05 perkfly

That is on the eventual to-do list!

benjaminfspector avatar May 16 '24 02:05 benjaminfspector

Great! Thanks for the brilliant project!

Are there any pointers on what the block issues are? I would like to try to fix this on my side.

来自手机回复

Benjamin Spector @.***>于2024年5月16日 周四10:33写道：

That is on the eventual to-do list!

— Reply to this email directly, view it on GitHub https://github.com/HazyResearch/ThunderKittens/issues/26#issuecomment-2113902174, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH4ZXZORXRM4YGACGHZGCLZCQLGNAVCNFSM6AAAAABHZHMMZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJTHEYDEMJXGQ . You are receiving this because you authored the thread.Message ID: @.***>

perkfly avatar May 16 '24 02:05 perkfly

looping in @Aaryan0404 for this

benjaminfspector avatar May 16 '24 19:05 benjaminfspector

Could you please explain why there is a limitation on the size of the head dimension? I'm not very clear about it.

That is on the eventual to-do list!

Could you please explain why there is a limitation on the size of the head dimension? I'm not very clear about it.

dz1iang avatar May 21 '24 12:05 dz1iang