SpectralNet
SpectralNet copied to clipboard
KlugerLab
Cholesky decomposition unsuccessful
Hi,
This has been asked before. I ran into the cholesky issue repeatedly in spite of trying large batch size. I wonder how is your experience of resolving this issue. Any tips would help, thank you in advance!
Lowering the learning rate helps as well. This occurs because the problem is a constrained convex optimization. If you go too fast then you can fly off the surface and get singularities.
On Wed, Aug 22, 2018 at 7:16 AM Yiyun Lan [email protected] wrote:
Hi,
This has been asked before. I ran into the cholesky issue repeatedly in spite of trying large batch size. I wonder how is your experience of resolving this issue. Any tips would help, thank you in advance!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/KlugerLab/SpectralNet/issues/11, or mute the thread https://github.com/notifications/unsubscribe-auth/AJXOK3HKFvbSD5JJmLKqgRyMVrfwnk2Yks5uTWfGgaJpZM4WHvyc .
Another tip is reducing the number of clusters, if possible.
One requirement of SpectralNet is that the orthonormalization layer is of rank equal to the number of clusters you set. Each minibatch must have enough variety / structure to have a full rank orthonormalization matrix. Thus, the dual to increasing the minibatch size is decreasing the cluster number. If your clusters are relatively balanced, and the number of clusters is on the order of a dozen or so, you're probably fine as is. But if it's much larger you might have problems. We have a few ideas in mind for loosening this restriction but there are no concrete plans yet.
Thank you for your input, that makes a lot sense. I was trying to get 30+ clusters in a fairly large dataset. And most likely it is not balanced.
I see. Yeah, this could be the reason why you had problems, especially if the classes are not balanced, unfortunately.
I find this problem in some datasets, such as FRGC.
Then I find it works when I change the epsilon (core/layers.py line 11) from 1e-7 to 1e-5 and reduce the spec_lr from 1e-3 to 1e-5.
