Plotting Issue with default HiGHS settings

[trevorb1]

Checklist

• [x] I am using the current master branch
• [x] I am running on an up-to-date pypsa-usa environment. Update via conda env update -f envs/environment.yaml

The Issue

I think there is a tolerance issue* with the HiGHS settings when reporting results. Some results will come out as negative and will cause stacked plots to break. If the solver is changed to Gurobi, the objective costs come out the same and the plotting is not an issue.

I originally thought it was just a tolerance thing, but looking at the timeseries results, the values are bigger than what I would expect for tolerance issues. Also, if you change the highs solving method from imp to simplex, the plotting no longer becomes an issue.

Steps To Reproduce

Im running this config:

scenario:
  interconnect: [western] #"usa|texas|western|eastern"
  clusters: [4m]
  simpl: [12]
  opts: [3h-REM]
  ll: [v1.0]
  sector: "" # G
  planning_horizons: [2030]    #(2018-2023, 2030, 2040, 2050)
foresight:  'perfect' # myopic, perfect

model_topology:
  transmission_network: 'reeds' # [reeds, tamu]
  topological_boundaries: 'reeds_zone' # [county, reeds_zone, state]
  interface_transmission_limits: false
  include: # mixed zone types not supported
    # reeds_zone: []
    reeds_state: ['WA']
    # reeds_ba: []
  aggregate: # eligible keys: [reeds_zone, trans_reg]
    # trans_grp: []
    # reeds_zone: []

...

  solver:
    name: highs
    options: highs-default

  solver_options:
    highs-default:
      # refer to https://ergo-code.github.io/HiGHS/options/definitions.html#solver
      threads: 4
      solver: "ipm"
      run_crossover: "off"
      small_matrix_value: 1e-6
      large_matrix_value: 1e9
      primal_feasibility_tolerance: 1e-5
      dual_feasibility_tolerance: 1e-5
      ipm_optimality_tolerance: 1e-4
      parallel: "on"
      random_seed: 123

Expected Behavior

No response

Error Message

Traceback (most recent call last):
  File "/home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmptwhseqi3.plot_statistics.py", line 993, in <module>
    plot_production_area(
  File "/home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmptwhseqi3.plot_statistics.py", line 553, in plot_production_area
    energy_mix.loc[sns].droplevel("period").round(2).plot.area(
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_core.py", line 1603, in area
    return self(kind="area", x=x, y=y, stacked=stacked, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_core.py", line 1030, in __call__
    return plot_backend.plot(data, kind=kind, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_matplotlib/__init__.py", line 71, in plot
    plot_obj.generate()
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_matplotlib/core.py", line 501, in generate
    self._make_plot(fig)
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_matplotlib/core.py", line 1544, in _make_plot
    newlines = plotf(
               ^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_matplotlib/core.py", line 1601, in _ts_plot
    lines = self._plot(ax, data.index, np.asarray(data.values), style=style, **kwds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_matplotlib/core.py", line 1753, in _plot
    y_values = cls._get_stacked_values(ax, stacking_id, y, kwds["label"])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/plotting/_matplotlib/core.py", line 1654, in _get_stacked_values
    raise ValueError(
ValueError: When stacked is True, each column must be either all positive or all negative. Column 'Combined-Cycle Gas' contains both positive and negative values

Anything else?

Here is where the issue pops up:

https://github.com/PyPSA/pypsa-usa/blob/76276718cafa48ab43df7bdb349bf54559094987/workflow/scripts/plot_statistics.py#L504-L573

And below is what the energy_mix variable looks like.