L2 Regularization for Sparsity?

[kayuksel]

L2 regularization as explained in your documentation penalizes large weights and hence would make your portfolio more diverse.

I don't think that it has a positive effect on eliminating negligible values. Another regularization method for having sparse portfolios:

https://www.sciencedirect.com/science/article/pii/S0378426619302614

[robertmartin8]

Hi @kayuksel,

Thanks for the suggestion! I'm afraid I don't understand the comment on L2 regularisation.

The idea behind L2 regularisation in PyPortfolioOpt is to make the portfolios less sparse rather than more sparse. If a user selects 20 stocks they like and passes them to PyPortfolioOpt, it can be irritating to see 15 zero weights (and unlikely to be optimal out-of-sample). The L2 reg helps here, pulling all weights towards an equal-weighted portfolio.

Open to suggestions though!

Best, Robert

[kayuksel]

Thank you very much for your response. I apologize for my misunderstanding. I thought that you were trying to make it more sparse and eliminate very low, negligible weights. I am working on the achieving the reverse since that I optimize thousands of asset portfolios rather than only 20. I personally hate seeing many very low non-zero values; of which cleaning may cause divergence in performance when you consider such a large number of assets. They are also problematic in terms of discrete allocation and requires very large portfolio to trade the resulting portfolio. You are definitely right that it would have lower generalization risk as a result of diversification. I would probably prefer using entropy rather than L2 for achieving the diversification though. By the way, what would you think about regularizing weights to have lower floating precision to later have better discrete allocations?

[robertmartin8]

No worries, you certainly bring up a valid feature request!

Part of the reason why haven't implemented any sparsity stuff is that I don't want to encourage people to use portfolio optimisation tools for portfolio selection, because I believe they are very different problems.

But this has been raised in several issues, so I am inclined to stop letting my philosophy get in people's way.

I agree that if you were trying to use portfolio optimisation for a large portfolio, small weights would be quite inconvenient. In the past, when I've had to optimise large portfolios I've normally used "hacks" like rounding away small weights, or removing the assets with low expected returns and high volatilities before optimising.

[kayuksel]

@robertmartin8 I have tried to learn how to sparsify via the following https://github.com/deep-spin/entmax

I actually want something where I can learn both sparsity and the temperature of a sparsemax function at the same time as using softmax (with temperature) seems to speed-up the portfolio weight optimization.

In terms of your philosophy, I believe that it is right since that current portfolio optimization methods are not good at selecting the optimum set from huge number of assets. This is actually what I am tackling.

[robertmartin8]

I wonder whether there's a quick solution involving CVXPY's built-in entropy atom