Tyler Kastner

icon indicating a website link tylerkastner.github.io icon indicating an email [email protected]

icon company University of Toronto, Vector Institute icon location Toronto icon location PhD Student at the University of Toronto

Results 1 issues of Tyler Kastner

Shouldn't we be dividing when normalizing QK^T, not multiplying?

In the code below, the query-key dot product is normalized by multiplying by the square root of the head size: https://github.com/karpathy/ng-video-lecture/blob/52201428ed7b46804849dea0b3ccf0de9df1a5c3/gpt.py#L83 Should we not be dividing instead? As seen in...