KFAC-Pytorch icon indicating copy to clipboard operation
KFAC-Pytorch copied to clipboard

Published 20 hours ago • verified publisher icon alecwangcq

KL Clipping may be error

Open zhengzangw opened this issue 1 year ago • 0 comments

https://github.com/alecwangcq/KFAC-Pytorch/blob/25e6dbe14752348d4f6030697b4b7f553ead2e92/optimizers/kfac.py#L137

The KL clipping should be $vFv^T$, here it seems to calculate $vF^{-1}v^T$.

zhengzangw avatar Oct 25 '23 12:10 zhengzangw