some runs of mpc end in an error : "ValueError: cannot reindex on an axis with duplicate labels "

[scruysberghs]

Everynow and then I get this error message

INFO - emhass.web_server - Passed runtime parameters: {'entity_save': True, 'publish_prefix': 'mpc_', 'prediction_horizon': 48, 'load_cost_forecast': [0.2641, 0.2641, 0.2623, 0.2623, 0.2638, 0.2638, 0.259, 0.259, 0.2819, 0.2819, 0.2998, 0.2998, 0.3019, 0.3019, 0.2809, 0.2809, 0.2656, 0.2656, 0.255, 0.255, 0.2397, 0.2397, 0.2368, 0.2368, 0.233, 0.233, 0.2273, 0.2273, 0.2234, 0.2234, 0.225, 0.225, 0.2367, 0.2367, 0.253, 0.253, 0.2893, 0.2893, 0.3257, 0.3257, 0.298, 0.298, 0.2758, 0.2758, 0.2602, 0.2602, 0.25, 0.25], 'prod_price_forecast': [0.1433, 0.1433, 0.1417, 0.1417, 0.1431, 0.1431, 0.1387, 0.1387, 0.1595, 0.1595, 0.1757, 0.1757, 0.1776, 0.1776, 0.1585, 0.1585, 0.1447, 0.1447, 0.1351, 0.1351, 0.1212, 0.1212, 0.1185, 0.1185, 0.1151, 0.1151, 0.1099, 0.1099, 0.1064, 0.1064, 0.1079, 0.1079, 0.1184, 0.1184, 0.1332, 0.1332, 0.1662, 0.1662, 0.1992, 0.1992, 0.1741, 0.1741, 0.1539, 0.1539, 0.1398, 0.1398, 0.1305, 0.1305], 'pv_power_forecast': [476, 560, 578, 506, 398, 278, 166, 68, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 15, 76, 138, 218, 319, 399, 446, 488, 530, 534, 508], 'soc_init': 0.12, 'soc_final': 0.3, 'alpha': 0, 'beta': 1, 'optimization_time_step': 30, 'historic_days_to_retrieve': 9, 'sensor_power_photovoltaics': 'sensor.sec_pac_solar', 'sensor_power_load_no_var_loads': 'sensor.sec_pac_non_deferable', 'sensor_linear_interp': ['sensor.sec_pac_non_deferable', 'sensor.sec_pac_solar'], 'num_def_loads': 4, 'operating_hours_of_each_deferrable_load': [0, 0, 0, 0], 'def_current_state': [False, False, False, False], 'nominal_power_of_deferrable_loads': [0.0, 0.0, 0.0, 0.0], 'start_timesteps_of_each_deferrable_load': [0, 0, 0, 0], 'end_timesteps_of_each_deferrable_load': [0, 0, 0, 0], 'treat_deferrable_load_as_semi_cont': [True, False, False, False], 'set_deferrable_load_single_constant': [True, False, False, False], 'set_deferrable_startup_penalty': [0, 0, 0, 310], 'maximum_power_from_grid': 3800, 'maximum_power_to_grid': 7000, 'inverter_is_hybrid': False, 'compute_curtailment': True, 'battery_discharge_power_max': 2500, 'battery_charge_power_max': 2500, 'battery_discharge_efficiency': 0.96, 'battery_charge_efficiency': 0.96, 'battery_nominal_energy_capacity': 5120, 'battery_maximum_state_of_charge': 0.98, 'battery_minimum_state_of_charge': 0.3, 'battery_target_state_of_charge': 0.3, 'weight_battery_discharge': 0.01, 'weight_battery_charge': 0.0, 'set_battery_dynamic': False, 'battery_dynamic_max': 0.9, 'battery_dynamic_min': -0.9, 'set_nocharge_from_grid': False, 'set_nodischarge_to_grid': False, 'costfun': 'profit', 'lp_solver': 'default', 'load_forecast_method': 'mlforecaster'}
INFO - emhass.web_server -  >> Setting input data dict
INFO - emhass.web_server - Setting up needed data
WARNING - emhass.web_server - Passed soc_init=0.12 is lower than soc_min=0.3, setting soc_init=soc_min
INFO - emhass.web_server - Retrieve hass get data method initiated...
INFO - emhass.web_server - Retrieving weather forecast data using method = list
INFO - emhass.web_server - Retrieving data from hass for load forecast using method = mlforecaster
INFO - emhass.web_server - Retrieve hass get data method initiated...
DEBUG - emhass.web_server - Number of ML predict forcast data generated (lags_opt): 48
DEBUG - emhass.web_server - Number of forcast dates obtained: 48
ERROR - emhass.web_server - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/web_server.py", line 414, in action_call
    input_data_dict = set_input_data_dict(
                      ^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/command_line.py", line 292, in set_input_data_dict
    df_input_data_dayahead = pd.concat([P_PV_forecast, P_load_forecast], axis=1)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/reshape/concat.py", line 395, in concat
    return op.get_result()
           ^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/reshape/concat.py", line 660, in get_result
    df = cons(data, index=index, copy=self.copy)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/frame.py", line 778, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/internals/construction.py", line 503, in dict_to_mgr
    return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/internals/construction.py", line 119, in arrays_to_mgr
    arrays, refs = _homogenize(arrays, index, dtype)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/internals/construction.py", line 611, in _homogenize
    val = val.reindex(index, copy=False)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/series.py", line 5153, in reindex
    return super().reindex(
           ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/generic.py", line 5610, in reindex
    return self._reindex_axes(
           ^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/generic.py", line 5633, in _reindex_axes
    new_index, indexer = ax.reindex(
                         ^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 4429, in reindex
    raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels

Then the next run, 5 minutes later, will work fine again. Any idea on where to look?

[han-alink]

Same behavior. It happens every hour (or every 12 time ticks of 5 minutes running mpc). Configured time step: 60 min. At the moment new data is available/retreived for the deferable loads.

[han-alink]

Add a delay of one minute gets rid of the errors:

triggers:

• minutes: /5 trigger: time_pattern actions:
• delay: minutes: 1

[MarcoPolet]

I have the same issue. I think I see it most often at xx:30 , I'm using nearest, so could be an 'edge case'. Thanks for the +1 minute hint, I'll try that as workaround.

[MarcoPolet]

Indeed the one minute delay works! Thanks!

[davidusb-geek]

Closing for now with the proposed delay solution. Re-open if needed