otoole icon indicating copy to clipboard operation
otoole copied to clipboard

Published 20 hours ago • verified publisher icon OSeMOSYS

[Bug]: `check_param_index_names` does not detect issues with case

Open willu47 opened this issue 1 year ago • 1 comments

The Issue

Headers for a parameters in an Excel file like this:

REGION | TECHNOLOGY | value

are not detected and another error is caused:

DEBUG:otoole.input:Expected indices for OperationalLife are ['REGION', 'TECHNOLOGY']
Traceback (most recent call last):
  File "/Users/wusher/miniconda3/envs/otoole38/bin/otoole", line 33, in <module>
    sys.exit(load_entry_point('otoole', 'console_scripts', 'otoole')())
  File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 470, in main
    args.func(args)
  File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 220, in conversion_matrix
    input_data, _ = read_strategy.read(args.from_path)
  File "/Users/wusher/repository/otoole/src/otoole/read_strategies.py", line 158, in read
    input_data = self._check_index(input_data)
  File "/Users/wusher/repository/otoole/src/otoole/input.py", line 388, in _check_index
    self._check_param_index_names(name=name, config=details, df=df)
  File "/Users/wusher/repository/otoole/src/otoole/input.py", line 435, in _check_param_index_names
    raise OtooleIndexError(
otoole.exceptions.OtooleIndexError: OperationalLife -> Indices inconsistent between config and data. Config indices are ['REGION', 'TECHNOLOGY']. Data indices are ['REGION', 'TECHNOLOGY', 'value', 'YEAR'].

Expected Behavior

An error should be raised stating that the index does not match the available sets, or that value should be upper case.

Steps To Reproduce

  1. In an Excel workbook, change the headers of a parameter from VALUE to value.
  2. Run otoole convert excel csv <workbook.xlsx> <test_folder> config.yaml

Log output

INFO:otoole.cli:Reading config from config.yaml
INFO:otoole.cli:Validating config from config.yaml
INFO:otoole.read_strategies:Checking set EMISSION
INFO:otoole.read_strategies:Checking set MODE_OF_OPERATION
INFO:otoole.read_strategies:Checking set REGION
INFO:otoole.read_strategies:Checking set FUEL
INFO:otoole.read_strategies:Checking set TIMESLICE
INFO:otoole.read_strategies:Checking set TECHNOLOGY
INFO:otoole.read_strategies:Checking set YEAR
INFO:otoole.read_strategies:AccumulatedAnnualDemand reshaped from wide to narrow format
INFO:otoole.read_strategies:AnnualEmissionLimit reshaped from wide to narrow format
INFO:otoole.read_strategies:AnnualExogenousEmission reshaped from wide to narrow format
INFO:otoole.read_strategies:AvailabilityFactor reshaped from wide to narrow format
INFO:otoole.read_strategies:CapacityFactor reshaped from wide to narrow format
INFO:otoole.read_strategies:CapacityToActivityUnit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:CapitalCost reshaped from wide to narrow format
INFO:otoole.read_strategies:DiscountRate is already in narrow form with headers ['REGION', 'VALUE']
INFO:otoole.read_strategies:DepreciationMethod is already in narrow form with headers ['REGION', 'VALUE']
INFO:otoole.read_strategies:EmissionActivityRatio reshaped from wide to narrow format
INFO:otoole.read_strategies:EmissionsPenalty reshaped from wide to narrow format
INFO:otoole.read_strategies:FixedCost reshaped from wide to narrow format
INFO:otoole.read_strategies:InputActivityRatio reshaped from wide to narrow format
INFO:otoole.read_strategies:OutputActivityRatio reshaped from wide to narrow format
INFO:otoole.read_strategies:ModelPeriodEmissionLimit is already in narrow form with headers ['REGION', 'EMISSION', 'VALUE']
INFO:otoole.read_strategies:ModelPeriodExogenousEmission is already in narrow form with headers ['REGION', 'EMISSION', 'VALUE']
INFO:otoole.read_strategies:OperationalLife reshaped from wide to narrow format
INFO:otoole.read_strategies:REMinProductionTarget reshaped from wide to narrow format
INFO:otoole.read_strategies:RETagFuel reshaped from wide to narrow format
INFO:otoole.read_strategies:RETagTechnology reshaped from wide to narrow format
INFO:otoole.read_strategies:ReserveMargin reshaped from wide to narrow format
INFO:otoole.read_strategies:ReserveMarginTagFuel reshaped from wide to narrow format
INFO:otoole.read_strategies:ReserveMarginTagTechnology reshaped from wide to narrow format
INFO:otoole.read_strategies:ResidualCapacity reshaped from wide to narrow format
INFO:otoole.read_strategies:SpecifiedAnnualDemand reshaped from wide to narrow format
INFO:otoole.read_strategies:SpecifiedDemandProfile reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMaxCapacity reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMaxCapacityInvestment reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMinCapacity reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMinCapacityInvestment reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalTechnologyAnnualActivityLowerLimit reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalTechnologyAnnualActivityUpperLimit reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalTechnologyModelPeriodActivityLowerLimit is already in narrow form with headers ['TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:TotalTechnologyModelPeriodActivityUpperLimit is already in narrow form with headers ['TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:TradeRoute reshaped from wide to narrow format
INFO:otoole.read_strategies:VariableCost reshaped from wide to narrow format
INFO:otoole.read_strategies:YearSplit reshaped from wide to narrow format
DEBUG:otoole.input:Identified EMISSION as a set
DEBUG:otoole.input:Identified MODE_OF_OPERATION as a set
DEBUG:otoole.input:Identified REGION as a set
DEBUG:otoole.input:Identified FUEL as a set
DEBUG:otoole.input:Identified TIMESLICE as a set
DEBUG:otoole.input:Identified TECHNOLOGY as a set
DEBUG:otoole.input:Identified YEAR as a set
DEBUG:otoole.input:Actual indices for AccumulatedAnnualDemand are ['REGION', 'FUEL', 'YEAR']
DEBUG:otoole.input:Expected indices for AccumulatedAnnualDemand are ['REGION', 'FUEL', 'YEAR']
DEBUG:otoole.input:Identified AccumulatedAnnualDemand as a parameter
DEBUG:otoole.input:Unable to set index on AccumulatedAnnualDemand
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'FUEL': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                        VALUE
REGION FUEL   YEAR           
NaN    BJFW3X 2015  76.263780
       BJHE3X 2015   0.000000
       BJHF3X 2015   0.690638
       BJLF3X 2015  68.373182
       BJCH3X 2015   0.000000
DEBUG:otoole.input:Actual indices for AnnualEmissionLimit are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Expected indices for AnnualEmissionLimit are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Identified AnnualEmissionLimit as a parameter
DEBUG:otoole.input:Unable to set index on AnnualEmissionLimit
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                      VALUE
REGION EMISSION YEAR       
NaN    BJREN    2015    0.0
       BFREN    2015    0.0
       CIREN    2015    0.0
       GMREN    2015    0.0
       GHREN    2015    0.0
DEBUG:otoole.input:Actual indices for AnnualExogenousEmission are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Expected indices for AnnualExogenousEmission are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Identified AnnualExogenousEmission as a parameter
DEBUG:otoole.input:Unable to set index on AnnualExogenousEmission
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:Empty DataFrame
Columns: [VALUE]
Index: []
DEBUG:otoole.input:Actual indices for AvailabilityFactor are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Expected indices for AvailabilityFactor are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Identified AvailabilityFactor as a parameter
DEBUG:otoole.input:Unable to set index on AvailabilityFactor
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                        VALUE
REGION TECHNOLOGY YEAR       
NaN    BJBMCHC01N 2015   0.93
       BJBMCHC02N 2015   0.93
       BJBMCHC03N 2015   0.93
       BJBMCHC04N 2015   0.93
       BJBMCHP01O 2015   0.93
DEBUG:otoole.input:Actual indices for CapacityFactor are ['REGION', 'TECHNOLOGY', 'TIMESLICE', 'YEAR']
DEBUG:otoole.input:Expected indices for CapacityFactor are ['REGION', 'TECHNOLOGY', 'TIMESLICE', 'YEAR']
DEBUG:otoole.input:Identified CapacityFactor as a parameter
DEBUG:otoole.input:Unable to set index on CapacityFactor
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'TIMESLICE': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                                  VALUE
REGION TECHNOLOGY TIMESLICE YEAR       
NaN    BJBMCHC01N S1D1      2015   0.54
                  S1D2      2015   0.54
                  S2D1      2015   0.54
                  S2D2      2015   0.54
                  S3D1      2015   0.54
DEBUG:otoole.input:Actual indices for CapacityToActivityUnit are ['REGION', 'TECHNOLOGY']
DEBUG:otoole.input:Expected indices for CapacityToActivityUnit are ['REGION', 'TECHNOLOGY']
DEBUG:otoole.input:Identified CapacityToActivityUnit as a parameter
DEBUG:otoole.input:Unable to set index on CapacityToActivityUnit
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:                    VALUE
REGION TECHNOLOGY        
NaN    BJBACKSTOP  31.536
       BJBMCHC01N  31.536
       BJBMCHC02N  31.536
       BJBMCHC03N  31.536
       BJBMCHC04N  31.536
DEBUG:otoole.input:Actual indices for CapitalCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Expected indices for CapitalCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Identified CapitalCost as a parameter
DEBUG:otoole.input:Unable to set index on CapitalCost
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                              VALUE
REGION TECHNOLOGY YEAR             
NaN    BJBACKSTOP 2015  99999.00000
       BJCRUDPROX 2015     24.12682
       BJNG00ILGX 2015      0.85500
       BJNG00ELGX 2015      1.52000
       BJBMCHC01N 2015   7808.00000
DEBUG:otoole.input:Actual indices for DiscountRate are ['REGION']
DEBUG:otoole.input:Expected indices for DiscountRate are ['REGION']
DEBUG:otoole.input:Identified DiscountRate as a parameter
DEBUG:otoole.input:Unable to set index on DiscountRate
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:        VALUE
REGION       
NaN       0.1
DEBUG:otoole.input:Actual indices for DepreciationMethod are ['REGION']
DEBUG:otoole.input:Expected indices for DepreciationMethod are ['REGION']
DEBUG:otoole.input:Identified DepreciationMethod as a parameter
DEBUG:otoole.input:Unable to set index on DepreciationMethod
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:        VALUE
REGION       
NaN         1
DEBUG:otoole.input:Actual indices for EmissionActivityRatio are ['REGION', 'TECHNOLOGY', 'EMISSION', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Expected indices for EmissionActivityRatio are ['REGION', 'TECHNOLOGY', 'EMISSION', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Identified EmissionActivityRatio as a parameter
DEBUG:otoole.input:Unable to set index on EmissionActivityRatio
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'EMISSION': 'str', 'MODE_OF_OPERATION': 'int', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                                                      VALUE
REGION TECHNOLOGY EMISSION MODE_OF_OPERATION YEAR          
NaN    BJCO00I00X BJCO2    1                 2015  0.090374
       BJCO00X00X BJCO2    1                 2015  0.090374
       BJLF00I00X BJCO2    1                 2015  0.069333
       BJHF00I00X BJCO2    1                 2015  0.074688
       BJCR00I00X BJCO2    1                 2015  0.070650
DEBUG:otoole.input:Actual indices for EmissionsPenalty are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Expected indices for EmissionsPenalty are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Identified EmissionsPenalty as a parameter
DEBUG:otoole.input:Unable to set index on EmissionsPenalty
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                      VALUE
REGION EMISSION YEAR       
NaN    BJCH4    2015    0.0
       BJN20    2015    0.0
       BJFGA    2015    0.0
       BJCO2    2015    0.0
       BFCH4    2015    0.0
DEBUG:otoole.input:Actual indices for FixedCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Expected indices for FixedCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Identified FixedCost as a parameter
DEBUG:otoole.input:Unable to set index on FixedCost
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                            VALUE
REGION TECHNOLOGY YEAR           
NaN    BJBACKSTOP 2015  999999.00
       BJETHANOLX 2015       0.00
       BJBMCHC01N 2015      91.28
       BJBMCHC02N 2015      91.28
       BJBMCHC03N 2015      91.28
DEBUG:otoole.input:Actual indices for InputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Expected indices for InputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Identified InputActivityRatio as a parameter
DEBUG:otoole.input:Unable to set index on InputActivityRatio
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'FUEL': 'str', 'MODE_OF_OPERATION': 'int', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                                                  VALUE
REGION TECHNOLOGY FUEL   MODE_OF_OPERATION YEAR        
NaN    BJUR00X00X BJWAT1 1                 2015  0.0012
       BJFW00P00X BJBIOM 1                 2015  1.0000
       BJCO00P00X BJCOAL 1                 2015  1.0000
       BJLF00P00X BJLFOI 1                 2015  1.0000
       BJHF00P00X BJHFOI 1                 2015  1.0000
DEBUG:otoole.input:Actual indices for OutputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Expected indices for OutputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Identified OutputActivityRatio as a parameter
DEBUG:otoole.input:Unable to set index on OutputActivityRatio
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'FUEL': 'str', 'MODE_OF_OPERATION': 'int', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:                                                 VALUE
REGION TECHNOLOGY FUEL   MODE_OF_OPERATION YEAR       
NaN    BJBACKSTOP BJEL01 1                 2015   1.00
       BJWA00000X BJWAT1 1                 2015   1.00
       BJFW00P00X BJFW3X 1                 2015   0.95
       BJCO00P00X BJCO3X 1                 2015   0.95
       BJLF00P00X BJLF3X 1                 2015   0.95
DEBUG:otoole.input:Actual indices for ModelPeriodEmissionLimit are ['REGION', 'EMISSION']
DEBUG:otoole.input:Expected indices for ModelPeriodEmissionLimit are ['REGION', 'EMISSION']
DEBUG:otoole.input:Identified ModelPeriodEmissionLimit as a parameter
DEBUG:otoole.input:Unable to set index on ModelPeriodEmissionLimit
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:Empty DataFrame
Columns: [VALUE]
Index: []
DEBUG:otoole.input:Actual indices for ModelPeriodExogenousEmission are ['REGION', 'EMISSION']
DEBUG:otoole.input:Expected indices for ModelPeriodExogenousEmission are ['REGION', 'EMISSION']
DEBUG:otoole.input:Identified ModelPeriodExogenousEmission as a parameter
DEBUG:otoole.input:Unable to set index on ModelPeriodExogenousEmission
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:Empty DataFrame
Columns: [VALUE]
Index: []
DEBUG:otoole.input:Actual indices for OperationalLife are ['REGION', 'TECHNOLOGY', 'value', 'YEAR']
DEBUG:otoole.input:Expected indices for OperationalLife are ['REGION', 'TECHNOLOGY']
Traceback (most recent call last):
  File "/Users/wusher/miniconda3/envs/otoole38/bin/otoole", line 33, in <module>
    sys.exit(load_entry_point('otoole', 'console_scripts', 'otoole')())
  File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 470, in main
    args.func(args)
  File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 220, in conversion_matrix
    input_data, _ = read_strategy.read(args.from_path)
  File "/Users/wusher/repository/otoole/src/otoole/read_strategies.py", line 158, in read
    input_data = self._check_index(input_data)
  File "/Users/wusher/repository/otoole/src/otoole/input.py", line 388, in _check_index
    self._check_param_index_names(name=name, config=details, df=df)
  File "/Users/wusher/repository/otoole/src/otoole/input.py", line 435, in _check_param_index_names
    raise OtooleIndexError(
otoole.exceptions.OtooleIndexError: OperationalLife -> Indices inconsistent between config and data. Config indices are ['REGION', 'TECHNOLOGY']. Data indices are ['REGION', 'TECHNOLOGY', 'value', 'YEAR'].

Operating System

MacOS

What version of otoole are you running?

v1.01

Possible Solution

This check in input.py could be modified, or an additional check for case could be added.

        if actual_indices == expected_indices:
            return
        else:
            raise OtooleIndexError(
                resource=name,
                config_indices=expected_indices,
                data_indices=actual_indices,
            )

Anything else?

No response

willu47 avatar Apr 20 '23 08:04 willu47