pytorch-hessian-eigenthings icon indicating copy to clipboard operation
pytorch-hessian-eigenthings copied to clipboard

Published 20 hours ago • verified publisher icon noahgolmant

Add continuous Hessian spectrum throughout training

Open noahgolmant opened this issue 6 years ago • 3 comments

Add GROUSE implementation for efficient low-rank subspace tracking with Grassmannian SGD based on the following paper (or other more recent subspace tracking papers): https://people.eecs.berkeley.edu/~brecht/papers/10.Bal.Now.Rec.GROUSE.pdf

noahgolmant avatar Nov 13 '18 00:11 noahgolmant

As a first step, just gonna try incrementally updating the eigenvector estimates using the previous computations as the initialization

noahgolmant avatar Nov 15 '18 12:11 noahgolmant

Regarding GROUSE, there seems to be a Matlab implementation by the author, not sure if that's going to be helpful. I'd be curious if this can be used in a similar way to the SVCCA paper by google, but I probably won't get to that.

themightyoarfish avatar Nov 20 '18 15:11 themightyoarfish

It's been a while since I thought about this feature, but that link is certainly helpful. Looking back on it, the GROUSE implementation could be modified to solve the least squares objective through the conjugate gradient method. I think this should end up having the same computational complexity as power iteration.

noahgolmant avatar Apr 01 '19 18:04 noahgolmant