sgd

sgd copied to clipboard

It's awful.

library(sgd)

# Dimensions
N <- 1e5
d <- 1e2

# Generate data.
X <- matrix(rnorm(N*d), ncol=d)
theta <- rep(5, d+1)
eps <- rnorm(N)
y <- cbind(1, X) %*% theta + eps
dat <- data.frame(y=y, x=X)

sgd.theta <- sgd(y ~ ., data=dat, model="lm", sgd.control=list(lr="one-dim-eigen"))
mean((sgd.theta$coefficients - theta)^2) # MSE
## [1] 24.7658

From intuition, I believe this is because our approximation to the upper bound on the minimal eigenvalue of the Fisher information is too noisy, and thus oftentimes not an upper bound:

λ_min ≤ trace(I_hat)/d

where the observed Fisher information I_hat is calculated by evaluating the score function at the data point in the current iteration, and taking the outer product with itself: \nabla l(θ_{n-1}; yn)*(\nabla l(θ_{n-1}; yn))^T. (See the learning rate wiki for more documentation.)

Therefore my guess is that the eigenvalues of the observed Fisher information here is unreliable.