Solver "OSQP" failed Try another solver. SCS doesn't work either with minimum weight portfolio optimization

[Porgen]

I am conducting an outof sample performance test for a minimum weight portfolio strategy and each time use the weights from max_sharpe() to revalue my portfolio. Minimum weight is 2%. There are 9 - 12 assets depending on the iteration. I keep running into this error if I use the SCS solver: pypfopt.exceptions.OptimizationError: ('Please check your objectives/constraints or use a different solver.', 'Solver status: infeasible')

OR OSQP failed if i simply use the default solver.

I have a weight training window of 60 or 90 days before valuing the portfolio over the subsequent 30 days. In all, I have 1230 days worth of asset data. A section of the code:

for t in range(60,len(df.iloc[0:1230])):
    k=t-60
  mu = expected_returns.mean_historical_return(df.iloc[t-60:t])
    S = CovarianceShrinkage(df.iloc[t-60:t]).ledoit_wolf()
    ef = EfficientFrontier(mu,S,solver="SCS",weight_bounds=(0.02,1))
    ef.max_sharpe()
    weights=pd.DataFrame(ef.clean_weights(),columns=ef.clean_weights().keys(), index=[0]).T
    weights=np.asarray(weights[0].to_list())

Is anyone able to assist?

[robertmartin8]

Do you know if it fails in every iteration of the loop? Or only in some. You can find out by doing a try/except within the loop.

I suggest you try to look at mu and S for the loop that fails: are there any NaNs or infinities?

[Porgen]

It fails 29 times in 1271 runs, and mu and S for those seem to be okay, which I found really strange.

[robertmartin8]

As a general suggestion for debugging, I would try to identify a loop where it fails and then recreate that exact code separately (outside of a loop).

If it fails, then you know the problem is in that particular chunk of the dataframe and you can explore for data issues.

If it succeeds, then the problem will be due to some weird caching behaviour of python / cvxpy

[Porgen]

Thanks Robert,  I managed to isolate the bit that struggles and it works perfectly outside the loop. Could you expand a bit more on the caching theory?

[robertmartin8]

Basically when you repeatedly solve optimisation problems in cvxpy (within the same script) it caches some data to make the next solve faster (Warm Start).

If that's causing problems, it can be disabled by passing solver_options={"warm_start":False} to the constructor for EfficientFrontier

[nematthews]

Hi there, I seem to be having a similar issue to Porgen. I have a loop that runs repeated EfficientFrontier max_sharpe() `optimizations with added sector constraints. It runs for several iterations but then produces the same optimization error message of 'Please check your objectives/constraints or use a different solver.', Solver status: infeasible'.

I removed the sector constraints within the loop and that seems to work, therefore I suspect it's related to these constraints. I have tried other optimization methods but still with no luck.

I then isolated the code outside of a loop and it runs successfully (with the sector constraints) for that single iteration. I've also tried passing solver_options={"warm_start":False} to EfficientFrontier within the loop but that still did not solve the issue.

Any input would be greatly appreciated. Thanks! @robertmartin8

[Porgen]

In my case, i changed my solver to mosek and that seemed to work. But you need a free and extendible license for this. Pyportfolioopt itself didn’t seem to solve this even after isolating the iterations to the problematic ones. strange