Unsuccessful mean-variance optimization in loop

[hirobank]

Describe the bug Hi there,

I was experiencing a very strange bug. When I used a loop to optimize the mean-variance weights, it seems randomly popping-up error messages. When retry the same input again, the problem disappears.

The bug only happens when tried to use statement ef.add_objective(objective_functions.L2_reg). I think probably there is some unintended assignment in the source code, which leads to unsuccessful release of the optimizer related instances.

I currently use the workaround rerunning the exact same input over and over again until it successes. Incredibly, it works...

I will be happy to provide you the full code if there is a non-public contact available.

========================================= Here is the code sample:

import pandas as pd import numpy as np from pypfopt.efficient_frontier import EfficientFrontier from pypfopt import objective_functions

[I use different cur_ret and shrink_cov in a loop] ef = EfficientFrontier(cur_ret, shrink_cov) ef.add_objective(objective_functions.L2_reg) weights = ef.min_volatility() del ef

[hirobank]

I saw a similar bug https://github.com/robertmartin8/PyPortfolioOpt/issues/436.

Unfortunately, when I changed it into ef = EfficientFrontier(cur_ret, shrink_cov, solver_options={"warm_start":False}) there were still numerous bugs.

[hirobank]

Hi there, I found the problem was caused by the wrong usage of mean-variance function. According to the document, the function should be max_quadratic_utility(). Thank you for supporting.

[mthelee]

Hi hirobank, I am working on a loop as well, and needa some expert view and guidence. May I have a sample of your code, how did you write a loop with different cur_ret and shrink_cov? Greatly appreciated.

[hirobank]

Hi mthelee,

Sorry for the late reply. I am happy to provide an example for you.

Basically, I put the below part into my code to run FOUR optimisations based on different data sources, where the loop was executed by month and year (cur_ym). Once the "try" catches an error, the current year and month (cur_ym) will be appended into a list to be executed again.

I set a "while" to exit when the list of errors has nothing. Trust me that it won't be infinite.

    try:
        ef = EfficientFrontier(None, shrink_cov)
        ef.add_constraint(lambda x: x >= 0.0)
        ef.add_constraint(lambda x: x <= 1.0)
        weights = ef.min_volatility()
        to_concat['seriesid'] = weights.keys()
        to_concat['cur_ym'] = cur_ym
        to_concat['sh_factor'] = shrinkage_f
        to_concat['min_var'] = weights.values()
        del ef

        ef = EfficientFrontier(cur_ret, shrink_cov)
        ef.add_constraint(lambda x: x >= 0.0)
        ef.add_constraint(lambda x: x <= 1.0)
        weights = ef.max_quadratic_utility()
        to_concat['mean_var'] = weights.values()
        del ef

        # Set the benchmark model
        past_sample_vol = cur_ret_df.loc[[i for i, x in enumerate(cur_ret_df['seriesid']) if x in shrink_cov.columns],
                          :].reset_index(drop=True).groupby('cur_YM').mean().std()
        shrink_cov = shrinkage_cov(lda_cov, ret_cov_12m, shrinkage_factor=shrinkage_f,
                                   mkt_vol=float(past_sample_vol))  # monthly basis

        ef = EfficientFrontier(cur_ret, shrink_cov)
        ef.add_constraint(lambda x: x >= 0.0)
        ef.add_constraint(lambda x: x <= 1.0)
        weights = ef.min_volatility()
        to_concat['min_var_bench'] = weights.values()
        del ef

        ef = EfficientFrontier(cur_ret, shrink_cov)
        ef.add_constraint(lambda x: x >= 0.0)
        ef.add_constraint(lambda x: x <= 1.0)
        weights = ef.max_quadratic_utility()
        to_concat['mean_var_bench'] = weights.values()
        del ef

        if cur_ym in error_list:
            print('Error solved in %d in shrinkage_factor %f' % (cur_ym, shrinkage_f))
            error_list.remove(cur_ym)
        return to_concat

    except:
        print('Error occured in %d in shrinkage_factor %f' % (cur_ym, shrinkage_f))
        if cur_ym not in error_list:
            error_list.append(cur_ym)
        return pd.DataFrame()