otoole [Bug]: Calculation of `CapitalInvestment` fails if `DiscountRateIdv` not defined in configuration file

The Issue

When running otoole to convert UTOPIA results calculated by CBC I get an error. When putting a print statement in result_package.py in line 315 for the DiscountRate the df seems to be empty.

Expected Behavior

In UTOPIA the DiscountRateIdv is not defined. Hence, otoole should take the DiscountRate.

Steps To Reproduce

No response

Log output

No response

Operating System

MacOS

What version of otoole are you running?

1.1.2.post1.dev25+gc3331f2

Possible Solution

No response

Anything else?

No response

Mar 14 '24 13:03 HauHe

The bug occurs when running otoole as part of OSeMOSYS_step

Mar 14 '24 13:03 HauHe

Hey @HauHe!

On your note of:

In UTOPIA the DiscountRateIdv is not defined. Hence, otoole should take the DiscountRate

is this is the actual expected behaviour? If DiscountRateIdv is not defined in the configuration file, then otoole wont know what value to use as default; regardless of how its defined in the model file. As a temporary solution, adding the definition to the otoole configuration file may fix the issue.

DiscountRateIdv:
    indices: [REGION, TECHNOLOGY]
    type: param
    dtype: float
    default: 0.05

Mar 14 '24 15:03 trevorb1

This does not get around the issue of this error not being correctly handled by otoole. Below is the log output if DiscountRateIdv is not in the configuration file:

otoole -v results cplex csv model.sol results datafile data.txt config.yaml

INFO:otoole.results.results:Looking for CapitalInvestment

Traceback (most recent call last):
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 1429, in setitem
    values[indexer] = casted
    ~~~~~~^^^^^^^^^
ValueError: could not broadcast input array from shape (0,) into shape (15,1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trevorb1/miniconda3/envs/step/bin/otoole", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/cli.py", line 318, in main
    args.func(args)
  File "/home/trevorb1/repos/otoole/src/otoole/cli.py", line 74, in _result_matrix
    convert_results(
  File "/home/trevorb1/repos/otoole/src/otoole/convert.py", line 139, in convert_results
    context.convert(from_path, to_path, input_data=input_data)
  File "/home/trevorb1/repos/otoole/src/otoole/input.py", line 113, in convert
    inputs, default_values = self._read(input_filepath, **kwargs)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/input.py", line 95, in _read
    return self._read_strategy.read(filepath, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/results/results.py", line 44, in read
    results = self.calculate_results(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/results/results.py", line 69, in calculate_results
    results[name] = results_package[name]
                    ~~~~~~~~~~~~~~~^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/results/result_package.py", line 96, in __getitem__
    results = self.result_mapper[name]()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/results/result_package.py", line 328, in capital_investment
    crf = capital_recovery_factor(
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/repos/otoole/src/otoole/results/result_package.py", line 798, in capital_recovery_factor
    crf[:] = values
    ~~~^^^
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/frame.py", line 4282, in __setitem__
    return self._setitem_slice(slc, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/frame.py", line 4306, in _setitem_slice
    self.iloc[key] = value
    ~~~~~~~~~^^^^^
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/indexing.py", line 911, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/indexing.py", line 1944, in _setitem_with_indexer
    self._setitem_single_block(indexer, value, name)
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/indexing.py", line 2218, in _setitem_single_block
    self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 415, in setitem
    return self.apply("setitem", indexer=indexer, value=value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 363, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 1432, in setitem
    raise ValueError(
ValueError: setting an array element with a sequence.

Mar 14 '24 15:03 trevorb1

Thanks @trevorb1 , indeed when adding the definition of DiscountRateIdv to the config file AND removing the default definition in the model file it runs.

Mar 14 '24 15:03 HauHe

I wonder if the issue is actually that in Utopia the DiscountRate is only defined via the setting the default to 0.5? Is otoole reading the default values from the text file? Or is it taking the default from the config file? Because in the function for calculating the capital_investment there is an if else condition, first checking if DiscountRateIdv is defined and if not it should take the DiscountRate but how come the DiscountRate df is empty when the default is defined in the config file?

Mar 14 '24 16:03 HauHe

adding the definition of DiscountRateIdv to the config file AND removing the default definition in the model file it runs.

When updating the configuration file, I left normal definition for discount rate and it ran fine:

param DiscountRate{r in REGION};
param DiscountRateIdv{r in REGION, t in TECHNOLOGY}, default DiscountRate[r];

However printed out these warning:

/home/trevorb1/repos/otoole/src/otoole/results/result_package.py:643: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  discounted_total_costs = discounted_operational_costs.add(
/home/trevorb1/repos/otoole/src/otoole/results/result_package.py:647: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  discounted_total_costs = discounted_total_costs.add(
/home/trevorb1/repos/otoole/src/otoole/results/result_package.py:651: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  discounted_total_costs = discounted_total_costs.sub(

Mar 14 '24 16:03 trevorb1

When updating the configuration file, I left normal definition for discount rate and it ran fine:

Interesting, that gave me an error.

(ose_step_dev) HauHe/OSeMOSYS_step > step --step_length 1 --step_length 5 --input_data data/utopia.txt --solver cbc Building and Solving Models: 0%| | 0/5 [00:00<?, ?it/s] Traceback (most recent call last): File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/bin/step", line 8, in <module> sys.exit(main()) ^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/haukeh/GitHub/HauHe/OSeMOSYS_step/src/osemosys_step/main.py", line 281, in main shutil.rmtree(item) File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/shutil.py", line 732, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/shutil.py", line 635, in _rmtree_safe_fd onerror(os.scandir, path, sys.exc_info()) File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/shutil.py", line 631, in _rmtree_safe_fd with os.scandir(topfd) as scandir_it: ^^^^^^^^^^^^^^^^^ NotADirectoryError: [Errno 20] Not a directory: PosixPath('results/.DS_Store')

Mar 14 '24 16:03 HauHe

Is this an otoole error?

Mar 14 '24 16:03 trevorb1

Not sure, but when removing the default from DiscountRateIdv in the model file it didn't occur again

Mar 14 '24 16:03 HauHe

Hmm... can you drop in the model/data files you are using, in addition to the otoole commands please? Seems weird that updating the DiscountRate all the sudden throws a NotADirectoryError. So just want to isolate the otoole issue from an osemosys_step issue!

Mar 14 '24 16:03 trevorb1