Liger-Kernel icon indicating copy to clipboard operation
Liger-Kernel copied to clipboard

Published 20 hours ago • verified publisher icon linkedin

Lenient Test

Open jon-chuang opened this issue 1 year ago • 1 comments

🐛 Describe the bug

https://github.com/linkedin/Liger-Kernel/blob/e249eee723978bf8610ff1ea2297d048a2417e20/test/transformers/test_swiglu.py#L46 https://github.com/linkedin/Liger-Kernel/blob/e249eee723978bf8610ff1ea2297d048a2417e20/test/transformers/test_geglu.py#L38

1e0 for fp32 and 1e4 for bf16

Seems a little excessive

If kernels don't cause models to diverge; this test ought to pass at much lower atols

Reproduce

No response

Versions

main

jon-chuang avatar Sep 03 '24 05:09 jon-chuang

Thanks! I think this is because of the scale of inputs. Can you try take a stab and see if we can lower it?

ByronHsu avatar Sep 03 '24 20:09 ByronHsu