kat
kat copied to clipboard
Adamdad
[ICLR2025] Kolmogorov-Arnold Transformer
I'm using KAT instead of normal attention. The validation losses came out with a difference of 0.000338227 between the runs.
I have successfully installed the kat_rational_cu module, but I’m still encountering an error when trying to use it. The error occurs during execution and points to a missing reference to...
Hi, I read in the paper that the gpu you guys are using is a single a5000 to train kat, what I am using is a single a6000, when I...
/home/anaconda3/bin/conda run -n kat --no-capture-output python /home/kat/validate.py /home/imagenet/ --model kat_tiny_patch16_224 --checkpoint kat_tiny_patch16_224_1f3ad3b2e69821f3d412f2924cf159a0e266f142d739cb68f68f796f5a0fe289.pth -b 512 /home/rational_kat_cu/kat_rational/kat_1dgroup.py:10: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead. @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32) /home/rational_kat_cu/kat_rational/kat_1dgroup.py:32: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated....
Hello. Thanks for sharing this amazing work. I believe there are some typos in paper. In particular, in the equation (13), the derivative $\frac{\partial P(x)}{\partial x}$ should be $\frac{\partial P(x)}{\partial...
when I use your kat_group as a module, it runs normally on cuda:0; however, when I run it on other GPUs, I encounter the error mentioned in the title. I...
Hi; The result from the transformer before the "forward_head" have shape [8, 197, 768] , however, if we would like to use it for segmentation, other model such as TransUnet...
KAT used the variance preserving initialization as formulated in the Kaimining initialization for learnable rational activations. This implies calculating the 2nd order moment of a rational function, which has a...
Hello author, when will you update the 2D version of KAN? I tried to modify it but failed.
