NTK4A icon indicating copy to clipboard operation
NTK4A copied to clipboard

Published 20 hours ago • verified publisher icon thegregyang
→

Metadata

Code for the paper: "Tensor Programs II: Neural Tangent Kernel for Any Architecture"

Results 2 NTK4A issues
Sort by recently updated
recently updated
newest added

About transformer attention scaling

Thank you very much for this great work! Regarding the calculation here: https://github.com/thegregyang/NTK4A/blob/master/Transformer-NTK.ipynb May I ask why the attention with key-query scaling $1/d_{head} = 1/n$ is used, instead of $1/\sqrt{d_{head}}$?

chenwydj avatar chenwydj

Implement BiRNN NTK Calculation (#1)

This PR implements calculating the BiRNN NTK using summation and concatenation on the BiRNN hidden states. The PR adds the python notebook, edits to utils, and NTK frob distance files...

santacml avatar santacml

← Metadata

96
Stars
10
Forks
Watchers

Owner

verified publisher icon thegregyang

Metadata

Code for the paper: "Tensor Programs II: Neural Tangent Kernel for Any Architecture"

Back