Sparse PCA

[sjaustirni]

(Related to #22)

Hey,

I would like to start implement sparse PCA and I have a question and need for feedback, as my ML experience is minute.

• Should we implement a separate algorithm (like scikit-learn does) or just use a sparse flag?
• By reading the sparse PCA paper linked in the original issue, the R implementation by the author, and this Python implementation, I have arrived at this (hyper)parameter list:
  • n_components (P)
  • l1_penalty (H)
  • l2_penalties(H)
  • max_n_iterations (P)
  • tolerance (P)
• I am not sure which of these should be parameters (P) and hyperparameters (H), feel free to comment on that or anything I missed here.

[bytesnake]

all of your items are hyperparameters, they are fixed prior to the training. The model parameter would be the eigen-decomposition. https://machinelearningmastery.com/difference-between-a-parameter-and-a-hyperparameter/ It would be awesome if you can re-use parts of the existing PCA infrastructure and fallback to the existing implementation if l1_penalty and l2_penalty are both zero. You can also use the elasticnet implementation in linfa-elasticnet.

[sjaustirni]

Hey @bytesnake thanks for your answer.

I was indeed confused about hyperparameters vs parameters, thank you for clearing it up :D

I want to, of course, use the existing PCA and ElasticNet implementation that we already have in linfa, especially since the linked Sparse PCA algorithm starts with an output from the ordinary PCA:

1. Let A start at V[,1:k], the loadings of the first k ordinary principal components.

~~I was wondering though whether the sparse PCA implementation should live in pca.rs under the sparse flag or whether SparsePCA should be a general wrapper around the existing PCA in its own class. I lean towards the latter, but I wanted to hear another opinion on this matter :)~~

Edit: I should figure that out from the penalty arguments, I did not get that first time around, but now I do :muscle: :D

[bytesnake]

awesome :tada: the current implementation of elasticnet only supports a single target, so you have to solve for every beta separately with Y=X alpha. Extension to multinomial regression can be found here https://statweb.stanford.edu/~jhf/ftp/noah.pdf