ArpackNoConvergence: ARPACK error -1: No convergence (301 iterations, 0/1 eigenvectors converged)

[jmie087]

Hello

I want to calculate a Markowitz portfolio while using exponentially-weighted mean and covariance. Since I wanted to use alpha rather than span, I wrote: alpha = 0.3 span_alpha = (2/alpha)-1 I wanted to try various alpha (from 0.1 to 0.9) and see the result.

My data has 30 assets. It consists of monthly historical price for 3 years.

ret = py.expected_returns.ema_historical_return(df, returns_data=True, compounding=False, span=span_alpha, frequency=12) cov = py.risk_models.exp_cov(df, returns_data=True, span=span_alpha, frequency=12) ef = EfficientFrontier(ret, cov, solver = cp.OSQP) ef.efficient_return(0.14)

The code works when alpha is 0.1 or 0.2. When alpha is bigger than 0.3, on the other hand, python returns error message: ArpackNoConvergence: ARPACK error -1: No convergence (301 iterations, 0/1 eigenvectors converged)

1. I tried differenct solver - cp.SCS and cp.ECOS. The result did not change.
2. I read https://github.com/cvxpy/cvxpy/issues/1424 So I tried to use psd_wrap(cov) but it did not work - TypeError: cov_matrix is not a dataframe or array Moreover, I used numpy.linalg.cholesky for a check and the covariance was PD.
3. I checked both my data and covariance to see if they have infinities or NaN. Neither the data nor the covariance had an issue.
4. I reduced the number of stocks from 30 to 20. Then the code seemed to work fine. But I really don't want to make my portfolio smaller. Is there any other way to solve the problem?

In addition, when I tried same optimization (30 stocks, alpha = 0.3) with excel solver, it worked fine.

[robertmartin8]

Thanks for raising this. I don't have any insights, especially since you say that the covariance matrix is PD.

That said, the TypeError is strange...

[jmie087]

Thanks for the comment! To explain TypeError problem in detail, what I meant was that ef = EfficientFrontier(ret, psd_wrap(cov), solver = cp.OSQP).efficient_return(0.14) caused the TypeError: cov_matrix is not a dataframe or array.

[robertmartin8]

What does psd_wrap(cov) look like?

[jmie087]

psd_wrap([[ 2.38725125e-02 2.49539523e-02 -2.25322677e-03 4.47205400e-03 6.74793082e-03 7.94803963e-03 6.37375483e-03 9.48619495e-03 8.69645386e-03 1.10628570e-03 2.16700391e-02 1.52893402e-04 1.19606609e-02 -1.86169182e-02 -5.32265814e-03 1.09739526e-02 9.87768619e-03 2.65777132e-02 2.82700514e-02 9.10025364e-04 1.08784748e-02 2.04140632e-02 1.35439395e-02 5.10712957e-03 -5.02802185e-03 5.01168599e-03 8.94280259e-03 5.27941293e-03 1.41027401e-02 1.37301440e-02] [ 2.49539523e-02 9.48714490e-02 -1.87001433e-02 9.29477126e-03 1.18727932e-02 2.56047856e-02 1.77122705e-02 3.43694428e-02 1.35557003e-02 7.63105614e-03 3.10866806e-02 1.35620908e-02 2.30036304e-02 -4.73764948e-02 1.42104177e-03 2.57078730e-02 3.71673810e-02 7.36565727e-02 4.47699253e-02 1.12520520e-03 2.26019422e-02 3.23823438e-02 3.12211104e-02 4.04293107e-02 -2.60625466e-03 3.47061389e-03 1.23995651e-02 1.37115833e-02 2.30918145e-02 1.77897641e-02] [-2.25322677e-03 -1.87001433e-02 1.84063822e-02 2.60742354e-03 -1.28665315e-03 -7.70890372e-03 -1.70393311e-03 4.12604343e-03 -9.43330325e-05 2.57550247e-03 -2.15153002e-03 4.84676475e-03 -3.63251585e-03 1.19957788e-02 5.12593715e-03 -6.62386954e-03 -3.72661508e-03 -1.52188494e-02 -6.00242797e-03 2.66660362e-03 -6.40643011e-03 5.68967255e-03 -9.87195026e-03 -8.47093789e-03 5.40251236e-03 7.55830445e-03 1.14378733e-03 -1.50576439e-03 -1.05021822e-02 3.10375943e-03] [ 4.47205400e-03 9.29477126e-03 2.60742354e-03 1.62185604e-02 1.02619964e-02 7.80426115e-03 8.24078948e-03 2.63895584e-02 1.40648705e-02 3.50160269e-03 -2.01693039e-03 1.16284885e-02 8.20335984e-03 -5.07025026e-03 6.25935504e-03 -4.59481362e-03 1.33697919e-02 3.05485652e-02 1.11916664e-02 7.64161220e-04 1.90003473e-03 2.53592774e-02 -3.90447222e-03 6.94802055e-03 1.24605421e-02 1.65677653e-02 1.05357803e-02 6.96205413e-03 -2.99468798e-03 3.42319348e-06] [ 6.74793082e-03 1.18727932e-02 -1.28665315e-03 1.02619964e-02 4.10964729e-02 9.35275603e-03 -3.99088993e-03 7.10136105e-03 1.41908194e-02 3.85419563e-03 1.28124321e-02 -6.87787016e-03 4.69932344e-03 5.23721323e-04 -1.77618116e-03 1.24866757e-02 -2.08887405e-05 1.75317160e-02 1.22721493e-02 -3.45077572e-03 1.89832232e-02 8.70320970e-03 1.18770201e-02 9.85742735e-03 -1.03180640e-02 1.91222310e-03 1.90278530e-03 2.74194700e-03 1.63408259e-02 -2.03602815e-03] [ 7.94803963e-03 2.56047856e-02 -7.70890372e-03 7.80426115e-03 9.35275603e-03 1.47678041e-02 8.33515674e-03 1.91792729e-02 8.72986141e-03 8.04652609e-04 5.46878639e-03 7.63670462e-03 7.68888312e-03 -1.54609799e-02 3.92392849e-03 1.92484174e-03 1.41880876e-02 3.04460338e-02 1.75770562e-02 -1.58158637e-03 6.68194876e-03 1.56037008e-02 1.17863439e-02 1.12246332e-02 3.98564759e-03 6.72311391e-03 6.63595420e-03 8.03573555e-03 6.11585211e-03 -2.85040668e-03] [ 6.37375483e-03 1.77122705e-02 -1.70393311e-03 8.24078948e-03 -3.99088993e-03 8.33515674e-03 1.99320322e-02 2.54422558e-02 7.09784866e-03 1.47261973e-03 8.73350099e-05 1.73092000e-02 8.25677244e-03 -3.20115975e-03 8.96184069e-03 -3.57864545e-03 1.89534229e-02 3.80348769e-02 1.04049866e-02 8.76138438e-04 -3.87638765e-03 1.87194893e-02 3.08856165e-03 5.62337122e-03 1.99368302e-02 1.33462944e-02 1.21712660e-02 9.54559090e-03 -4.72903347e-03 2.14321740e-03] [ 9.48619495e-03 3.43694428e-02 4.12604343e-03 2.63895584e-02 7.10136105e-03 1.91792729e-02 2.54422558e-02 7.91995235e-02 1.50072004e-02 1.06201395e-02 -4.49007054e-03 4.25814426e-02 9.76063233e-03 1.24032239e-02 2.93875644e-02 -3.34592162e-03 4.31444411e-02 7.55999615e-02 3.45143702e-02 -7.70201039e-04 -7.99477246e-03 6.09323393e-02 -6.17556269e-03 1.91734217e-02 3.82190968e-02 3.55824470e-02 1.89988100e-02 1.62797070e-02 -1.22111491e-02 1.82847147e-03] [ 8.69645386e-03 1.35557003e-02 -9.43330325e-05 1.40648705e-02 1.41908194e-02 8.72986141e-03 7.09784866e-03 1.50072004e-02 2.33649853e-02 2.70515846e-03 7.61998030e-03 5.28471374e-03 1.34635578e-02 -2.70054719e-02 1.88800081e-03 -4.90280356e-03 7.20724141e-03 3.45716024e-02 8.25350085e-03 1.05764840e-03 9.42384816e-03 1.93769480e-02 7.57629131e-03 8.75943034e-03 4.75798260e-03 1.20014182e-02 1.49255431e-02 7.14456046e-03 1.54351919e-03 1.01470400e-03] [ 1.10628570e-03 7.63105614e-03 2.57550247e-03 3.50160269e-03 3.85419563e-03 8.04652609e-04 1.47261973e-03 1.06201395e-02 2.70515846e-03 1.03673453e-02 3.40549117e-03 2.93103665e-03 1.97466553e-03 4.07895749e-03 3.42898055e-04 4.82824826e-03 5.93079180e-03 1.41480711e-02 2.37336152e-03 2.58390250e-03 2.24036692e-03 1.06241293e-02 7.89838599e-03 6.87264715e-03 8.41168496e-03 7.31092070e-03 1.90758208e-03 1.07758984e-03 -2.28638609e-04 3.35475163e-03] [ 2.16700391e-02 3.10866806e-02 -2.15153002e-03 -2.01693039e-03 1.28124321e-02 5.46878639e-03 8.73350099e-05 -4.49007054e-03 7.61998030e-03 3.40549117e-03 3.79701951e-02 -5.91790962e-03 1.30288252e-02 -2.25681487e-02 -1.11830040e-02 1.91565853e-02 1.52713849e-03 1.96806867e-02 2.17377226e-02 2.98301105e-03 1.97179591e-02 9.07720430e-03 2.67467130e-02 6.28808636e-03 -1.11902051e-02 -4.44294282e-03 6.67294786e-03 5.48018385e-03 2.17916530e-02 1.81166623e-02] [ 1.52893402e-04 1.35620908e-02 4.84676475e-03 1.16284885e-02 -6.87787016e-03 7.63670462e-03 1.73092000e-02 4.25814426e-02 5.28471374e-03 2.93103665e-03 -5.91790962e-03 4.06335401e-02 6.99024492e-04 9.60757748e-03 1.51875927e-02 -1.65146001e-02 2.70834045e-02 3.52114136e-02 8.30769074e-03 2.61438606e-03 -1.43418488e-02 2.90653479e-02 -6.50913367e-03 1.35086111e-03 3.19499325e-02 1.84324366e-02 9.08690834e-03 7.91182824e-03 -1.61998206e-02 -3.44723927e-03] [ 1.19606609e-02 2.30036304e-02 -3.63251585e-03 8.20335984e-03 4.69932344e-03 7.68888312e-03 8.25677244e-03 9.76063233e-03 1.34635578e-02 1.97466553e-03 1.30288252e-02 6.99024492e-04 1.53942690e-02 -2.24983287e-02 -3.11263366e-03 5.00266984e-03 9.80296063e-03 3.15427150e-02 1.37965699e-02 2.76878084e-03 9.89830959e-03 1.59094865e-02 1.04380518e-02 1.05186858e-02 2.49400168e-03 7.32319064e-03 1.24963328e-02 8.03209413e-03 7.19782061e-03 8.65009353e-03] [-1.86169182e-02 -4.73764948e-02 1.19957788e-02 -5.07025026e-03 5.23721323e-04 -1.54609799e-02 -3.20115975e-03 1.24032239e-02 -2.70054719e-02 4.07895749e-03 -2.25681487e-02 9.60757748e-03 -2.24983287e-02 1.49334248e-01 -7.36160602e-03 2.05402748e-02 4.00520427e-03 -2.19691097e-02 -1.78106931e-02 9.03664459e-04 -1.73631137e-02 -1.19619834e-02 -3.15264160e-02 -2.79467860e-02 3.41331323e-02 1.10127037e-03 -1.26736118e-02 1.13383090e-03 -5.23358251e-03 1.06016297e-03] [-5.32265814e-03 1.42104177e-03 5.12593715e-03 6.25935504e-03 -1.77618116e-03 3.92392849e-03 8.96184069e-03 2.93875644e-02 1.88800081e-03 3.42898055e-04 -1.11830040e-02 1.51875927e-02 -3.11263366e-03 -7.36160602e-03 3.67860528e-02 -1.05824275e-02 8.37683347e-03 1.53682899e-02 7.44167408e-03 -7.24513430e-03 -1.16486991e-02 1.67315559e-02 -1.05088952e-02 9.26218598e-03 5.47239697e-03 1.02575154e-02 5.27095780e-03 8.11971596e-04 -1.67462976e-02 -5.49045453e-03] [ 1.09739526e-02 2.57078730e-02 -6.62386954e-03 -4.59481362e-03 1.24866757e-02 1.92484174e-03 -3.57864545e-03 -3.34592162e-03 -4.90280356e-03 4.82824826e-03 1.91565853e-02 -1.65146001e-02 5.00266984e-03 2.05402748e-02 -1.05824275e-02 3.82486643e-02 2.37587098e-03 1.22626126e-02 1.90096928e-02 -7.29710936e-04 1.65893087e-02 2.29404653e-04 8.36464122e-03 1.11045047e-02 -1.34402804e-02 -9.80618286e-03 -2.66882449e-03 2.46577396e-03 2.39915222e-02 1.83120661e-02] [ 9.87768619e-03 3.71673810e-02 -3.72661508e-03 1.33697919e-02 -2.08887405e-05 1.41880876e-02 1.89534229e-02 4.31444411e-02 7.20724141e-03 5.93079180e-03 1.52713849e-03 2.70834045e-02 9.80296063e-03 4.00520427e-03 8.37683347e-03 2.37587098e-03 3.40887831e-02 5.27392946e-02 2.36109633e-02 2.18257939e-03 -1.94384966e-03 3.37545862e-02 3.36550728e-03 1.53835375e-02 2.31526332e-02 1.70635609e-02 1.14637378e-02 1.10876317e-02 -1.42707511e-03 4.95199194e-03] [ 2.65777132e-02 7.36565727e-02 -1.52188494e-02 3.05485652e-02 1.75317160e-02 3.04460338e-02 3.80348769e-02 7.55999615e-02 3.45716024e-02 1.41480711e-02 1.96806867e-02 3.52114136e-02 3.15427150e-02 -2.19691097e-02 1.53682899e-02 1.22626126e-02 5.27392946e-02 1.36697098e-01 4.86045762e-02 1.55347066e-03 1.12114454e-02 6.69143879e-02 2.29083573e-02 3.48215567e-02 3.68153141e-02 3.13907815e-02 3.39548833e-02 2.50630922e-02 1.04412233e-02 1.39263151e-02] [ 2.82700514e-02 4.47699253e-02 -6.00242797e-03 1.11916664e-02 1.22721493e-02 1.75770562e-02 1.04049866e-02 3.45143702e-02 8.25350085e-03 2.37336152e-03 2.17377226e-02 8.30769074e-03 1.37965699e-02 -1.78106931e-02 7.44167408e-03 1.90096928e-02 2.36109633e-02 4.86045762e-02 5.17351328e-02 -2.74462974e-03 1.12815528e-02 4.04458434e-02 1.16581595e-02 1.65012610e-02 -4.87765908e-03 1.05671111e-02 9.90731110e-03 9.29405785e-03 1.77480408e-02 1.54360989e-02] [ 9.10025364e-04 1.12520520e-03 2.66660362e-03 7.64161220e-04 -3.45077572e-03 -1.58158637e-03 8.76138438e-04 -7.70201039e-04 1.05764840e-03 2.58390250e-03 2.98301105e-03 2.61438606e-03 2.76878084e-03 9.03664459e-04 -7.24513430e-03 -7.29710936e-04 2.18257939e-03 1.55347066e-03 -2.74462974e-03 5.00627622e-03 -3.21935250e-04 1.34439175e-03 2.91581036e-03 -1.57426452e-03 5.60026041e-03 2.52610555e-03 1.46936230e-03 1.17366851e-03 -4.50065415e-04 3.36266984e-03] [ 1.08784748e-02 2.26019422e-02 -6.40643011e-03 1.90003473e-03 1.89832232e-02 6.68194876e-03 -3.87638765e-03 -7.99477246e-03 9.42384816e-03 2.24036692e-03 1.97179591e-02 -1.43418488e-02 9.89830959e-03 -1.73631137e-02 -1.16486991e-02 1.65893087e-02 -1.94384966e-03 1.12114454e-02 1.12815528e-02 -3.21935250e-04 2.14037740e-02 1.23809681e-03 1.78759457e-02 9.80915821e-03 -1.30066935e-02 -4.07720790e-03 3.11997494e-03 3.75864662e-03 1.91401469e-02 6.90790822e-03] [ 2.04140632e-02 3.23823438e-02 5.68967255e-03 2.53592774e-02 8.70320970e-03 1.56037008e-02 1.87194893e-02 6.09323393e-02 1.93769480e-02 1.06241293e-02 9.07720430e-03 2.90653479e-02 1.59094865e-02 -1.19619834e-02 1.67315559e-02 2.29404653e-04 3.37545862e-02 6.69143879e-02 4.04458434e-02 1.34439175e-03 1.23809681e-03 6.54303939e-02 -2.48500447e-04 1.64596518e-02 2.53543917e-02 3.40913620e-02 2.06724799e-02 1.41537866e-02 -2.72408045e-03 1.05978913e-02] [ 1.35439395e-02 3.12211104e-02 -9.87195026e-03 -3.90447222e-03 1.18770201e-02 1.17863439e-02 3.08856165e-03 -6.17556269e-03 7.57629131e-03 7.89838599e-03 2.67467130e-02 -6.50913367e-03 1.04380518e-02 -3.15264160e-02 -1.05088952e-02 8.36464122e-03 3.36550728e-03 2.29083573e-02 1.16581595e-02 2.91581036e-03 1.78759457e-02 -2.48500447e-04 4.86182745e-02 9.50239817e-03 -2.41568284e-03 -1.42926138e-03 3.62525001e-03 5.65762777e-03 1.79572224e-02 1.04472039e-03] [ 5.10712957e-03 4.04293107e-02 -8.47093789e-03 6.94802055e-03 9.85742735e-03 1.12246332e-02 5.62337122e-03 1.91734217e-02 8.75943034e-03 6.87264715e-03 6.28808636e-03 1.35086111e-03 1.05186858e-02 -2.79467860e-02 9.26218598e-03 1.11045047e-02 1.53835375e-02 3.48215567e-02 1.65012610e-02 -1.57426452e-03 9.80915821e-03 1.64596518e-02 9.50239817e-03 3.08178160e-02 -6.56124960e-03 2.94632268e-03 6.10239506e-03 3.68422950e-03 5.49737641e-03 4.46434322e-03] [-5.02802185e-03 -2.60625466e-03 5.40251236e-03 1.24605421e-02 -1.03180640e-02 3.98564759e-03 1.99368302e-02 3.82190968e-02 4.75798260e-03 8.41168496e-03 -1.11902051e-02 3.19499325e-02 2.49400168e-03 3.41331323e-02 5.47239697e-03 -1.34402804e-02 2.31526332e-02 3.68153141e-02 -4.87765908e-03 5.60026041e-03 -1.30066935e-02 2.53543917e-02 -2.41568284e-03 -6.56124960e-03 5.13651952e-02 2.65317238e-02 1.22441178e-02 1.30716745e-02 -1.72190703e-02 -4.49243713e-03] [ 5.01168599e-03 3.47061389e-03 7.55830445e-03 1.65677653e-02 1.91222310e-03 6.72311391e-03 1.33462944e-02 3.55824470e-02 1.20014182e-02 7.31092070e-03 -4.44294282e-03 1.84324366e-02 7.32319064e-03 1.10127037e-03 1.02575154e-02 -9.80618286e-03 1.70635609e-02 3.13907815e-02 1.05671111e-02 2.52610555e-03 -4.07720790e-03 3.40913620e-02 -1.42926138e-03 2.94632268e-03 2.65317238e-02 2.72224245e-02 1.31674076e-02 9.20895154e-03 -1.08576706e-02 -1.02026379e-03] [ 8.94280259e-03 1.23995651e-02 1.14378733e-03 1.05357803e-02 1.90278530e-03 6.63595420e-03 1.21712660e-02 1.89988100e-02 1.49255431e-02 1.90758208e-03 6.67294786e-03 9.08690834e-03 1.24963328e-02 -1.26736118e-02 5.27095780e-03 -2.66882449e-03 1.14637378e-02 3.39548833e-02 9.90731110e-03 1.46936230e-03 3.11997494e-03 2.06724799e-02 3.62525001e-03 6.10239506e-03 1.22441178e-02 1.31674076e-02 1.60590995e-02 9.60568437e-03 -2.00703572e-03 5.29035441e-03] [ 5.27941293e-03 1.37115833e-02 -1.50576439e-03 6.96205413e-03 2.74194700e-03 8.03573555e-03 9.54559090e-03 1.62797070e-02 7.14456046e-03 1.07758984e-03 5.48018385e-03 7.91182824e-03 8.03209413e-03 1.13383090e-03 8.11971596e-04 2.46577396e-03 1.10876317e-02 2.50630922e-02 9.29405785e-03 1.17366851e-03 3.75864662e-03 1.41537866e-02 5.65762777e-03 3.68422950e-03 1.30716745e-02 9.20895154e-03 9.60568437e-03 1.07150345e-02 2.18153855e-03 2.37853142e-03] [ 1.41027401e-02 2.30918145e-02 -1.05021822e-02 -2.99468798e-03 1.63408259e-02 6.11585211e-03 -4.72903347e-03 -1.22111491e-02 1.54351919e-03 -2.28638609e-04 2.17916530e-02 -1.61998206e-02 7.19782061e-03 -5.23358251e-03 -1.67462976e-02 2.39915222e-02 -1.42707511e-03 1.04412233e-02 1.77480408e-02 -4.50065415e-04 1.91401469e-02 -2.72408045e-03 1.79572224e-02 5.49737641e-03 -1.72190703e-02 -1.08576706e-02 -2.00703572e-03 2.18153855e-03 2.67008445e-02 9.94152759e-03] [ 1.37301440e-02 1.77897641e-02 3.10375943e-03 3.42319348e-06 -2.03602815e-03 -2.85040668e-03 2.14321740e-03 1.82847147e-03 1.01470400e-03 3.35475163e-03 1.81166623e-02 -3.44723927e-03 8.65009353e-03 1.06016297e-03 -5.49045453e-03 1.83120661e-02 4.95199194e-03 1.39263151e-02 1.54360989e-02 3.36266984e-03 6.90790822e-03 1.05978913e-02 1.04472039e-03 4.46434322e-03 -4.49243713e-03 -1.02026379e-03 5.29035441e-03 2.37853142e-03 9.94152759e-03 2.24272833e-02]])

This is the result I got from psd_wrap(cov) when alpha is 0.2.

More over, by print(type(psd_wrap(cov))), I got <class 'cvxpy.atoms.affine.wraps.psd_wrap'>. By print(type(cov)), I got <class 'pandas.core.frame.DataFrame'>.

[robertmartin8]

Ah - that explains the type error. You are passing a cvxpy object to a function that expects an array/df. I think in cvxpy you can extract the array from the object but you’ll have to check their docs as I can’t remember how