pypsa-usa
pypsa-usa copied to clipboard
PyPSA
Error when setting sector option to "G"
Checklist
- [x] I am using the current
masterbranch - [x] I am running on an up-to-date
pypsa-usaenvironment. Update viaconda env update -f envs/environment.yaml
The Issue
When setting the sector option to "G" (in the config.default.yaml file), the following error occurs:
Traceback (most recent call last):
File "/home/dummy/workspace/pypsa-usa/workflow/.snakemake/scripts/tmppyu6l9uz.solve_network.py", line 1912, in <module>
n.export_to_netcdf(snakemake.output[0])
File "/home/dummy/workspace/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pypsa/io.py", line 816, in export_to_netcdf
_export_to_exporter(
File "/home/dummy/workspace/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pypsa/io.py", line 583, in _export_to_exporter
col_export = pnl[attr].columns
^^^^^^^^^^^^^^^^^
File "/home/dummy/workspace/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/generic.py", line 6299, in __getattr__
return object.__getattribute__(self, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Series' object has no attribute 'columns'
Steps To Reproduce
No response
Expected Behavior
No response
Error Message
Anything else?
No response
@ricnogfer To reproduce, are you able to drop in your scenario, foresight, and model_topology settings from the configuration file, please!
Thanks for looking into it @trevorb1! Here is the content of the config.default.yaml file:
# PyPSA-USA Default Configuration File for developing and testing county transmission network from Reeds
run:
name: "demo" # use this to keep track of runs with different settings
disable_progressbar: false # set to true to disable the progressbar
shared_resources: true # set to true to share the default resources across runs
shared_cutouts: true # set to true to share the default cutout(s) across runs
validation: false # set to true to run back-casting plots
# docs :
scenario:
interconnect: [western] #"usa|texas|western|eastern"
clusters: [33]
simpl: [75]
opts: [REM-336h]
ll: [v1.0]
sector: "G" # G
planning_horizons: [2030] #(2018-2023, 2030, 2040, 2050)
foresight: 'perfect' # myopic, perfect
model_topology:
transmission_network: 'reeds' # [reeds, tamu]
topological_boundaries: 'reeds_zone' # [county, reeds_zone]
interface_transmission_limits: false
include: # mixed zone types not supported
# reeds_zone: []
# reeds_state: ['CA']
# reeds_ba: []
aggregate: # eligible keys: [reeds_zone, trans_reg]
# trans_grp: []
# reeds_zone: []
# docs :
enable:
build_cutout: false
renewable_weather_years: [2019]
snapshots:
start: "2019-01-01"
end: "2020-01-01"
inclusive: "left"
# docs :
electricity:
conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal, biomass, waste] # Choose the conventional plant types to include in network
renewable_carriers: [onwind, offwind_floating, solar, hydro] # Choose the renewable plant types to include in network
retirement: economic # "economic" or "technical"
SAFE_reservemargin: 0.14
regional_Co2_limits: 'config/policy_constraints/regional_Co2_limits.csv'
technology_capacity_targets: 'config/policy_constraints/technology_capacity_targets.csv'
portfolio_standards: 'config/policy_constraints/portfolio_standards.csv'
SAFE_regional_reservemargins: 'config/policy_constraints/SAFE_regional_prm.csv'
transmission_interface_limits: 'config/policy_constraints/transmission_interface_limits.csv'
operational_reserve:
activate: false
epsilon_load: 0.02
epsilon_vres: 0.02
contingency: 4000
extendable_carriers:
Generator: [solar, onwind, offwind_floating, OCGT, CCGT, CCGT-95CCS, coal, nuclear, hydrogen_ct] #include CCGT-CCS
StorageUnit: [4hr_battery_storage, 8hr_battery_storage] # [Xhr-battery-storage (2-10 hours)]
Store: []
Link: []
demand:
profile: efs # efs, eia
scenario:
efs_case: reference # reference, medium, high
efs_speed: moderate # slow, moderate, rapid
aeo: reference
demand_response:
shift: 0
marginal_cost: 999999
# docs :
conventional:
unit_commitment: false
must_run: false
dynamic_fuel_price:
enable: false
pudl: true
wholesale: true
# docs :
lines:
s_max_pu: 0.7
s_nom_max: .inf
max_extension: 20000 #MW
length_factor: 1.25
# docs :
links:
p_max_pu: 1.0
p_nom_max: .inf
max_extension: 20000 #MW
# docs :
costs:
atb:
model_case: "Market" # Market, R&D
scenario: "Moderate" # Advanced, Conservative, Moderate
aeo:
scenario: "reference" # reference, high, low
social_discount_rate: 0.02
ng_fuel_year: 2019 # year of the natural gas price from CAISO [2019- 2023]
emission_prices: # in currency per tonne emission, only used with the option Ep
enable: false
co2: 0.
co2_monthly_prices: false
ptc_modifier:
onwind: 27.50
biomass: 27.50
itc_modifier:
solar: 0.3
offwind: 0.3
offwind_floating: 0.3
EGS: 0.3
geothermal: 0.3
SMR: 0.3
nuclear: 0.3
hydro: 0.3
2hr_battery_storage: 0.3
4hr_battery_storage: 0.3
6hr_battery_storage: 0.3
8hr_battery_storage: 0.3
10hr_battery_storage: 0.3
8hr_PHS: 0.3
10hr_PHS: 0.3
12hr_PHS: 0.3
min_year: # {carrier: year}
hydrogen_ct: 2040
max_growth: # {carrier: {base:, rate:}}
# docs :
clustering:
simplify_network:
weighting_strategy: demand-capacity # choose from: [population, demand-capacity]
to_substations: false # network is simplified to nodes with positive or negative power injection (i.e. substations or offwind connections)
algorithm: kmeans # choose from: [hac, kmeans]
feature: solar+onwind-time # only for hac. choose from: [solar+onwind-time, solar+onwind-cap, solar-time, solar-cap, solar+offwind-cap] etc.
cluster_network:
weighting_strategy: demand-capacity # choose from: [population, demand-capacity]
algorithm: kmeans # choose from: [hac, kmeans]
feature: solar+onwind-time
exclude_carriers: []
consider_efficiency_classes: false
aggregation_strategies:
generators:
build_year: 'capacity_weighted_average'
lifetime: 'capacity_weighted_average'
start_up_cost: 'capacity_weighted_average'
min_up_time: 'capacity_weighted_average'
min_down_time: 'capacity_weighted_average'
ramp_limit_up: max
ramp_limit_down: max
committable: any
vom_cost: mean
fuel_cost: mean
heat_rate: mean
temporal:
resolution_elec: false
resolution_sector: false
focus_weights:
# docs :
solving:
#tmpdir: "path/to/tmp"
options:
load_shedding: false
clip_p_max_pu: 1.e-2
noisy_costs: true
skip_iterations: true
rolling_horizon: false
seed: 123
# options that go into the optimize function
track_iterations: false
min_iterations: 4
max_iterations: 6
transmission_losses: 2
linearized_unit_commitment: true
horizon: 8760
assign_all_duals: true
solver:
name: gurobi
options: gurobi-default
solver_options:
highs-default:
# refer to https://ergo-code.github.io/HiGHS/options/definitions.html#solver
threads: 4
solver: "ipm"
run_crossover: "off"
small_matrix_value: 1e-6
large_matrix_value: 1e9
primal_feasibility_tolerance: 1e-5
dual_feasibility_tolerance: 1e-5
ipm_optimality_tolerance: 1e-4
parallel: "on"
random_seed: 123
gurobi-default:
threads: 8
method: 2 # barrier
crossover: 0
BarHomogeneous: 1 # Use homogeneous barrier if standard does not converge
BarConvTol: 1.e-5
OptimalityTol: 1.e-4
FeasibilityTol: 1.e-3
ScaleFlag: 1
Seed: 123
AggFill: 0
PreDual: 0
GURO_PAR_BARDENSETHRESH: 200
gurobi-numeric-focus:
name: gurobi
NumericFocus: 3 # Favour numeric stability over speed
method: 2 # barrier
crossover: 0 # do not use crossover
BarHomogeneous: 1 # Use homogeneous barrier if standard does not converge
BarConvTol: 1.e-5
FeasibilityTol: 1.e-4
OptimalityTol: 1.e-4
ObjScale: -0.5
threads: 8
Seed: 123
gurobi-fallback: # Use gurobi defaults
name: gurobi
crossover: 0
method: 2 # barrier
BarHomogeneous: 1 # Use homogeneous barrier if standard does not converge
BarConvTol: 1.e-5
FeasibilityTol: 1.e-5
OptimalityTol: 1.e-5
Seed: 123
threads: 8
cplex-default:
threads: 4
lpmethod: 4 # barrier
solutiontype: 2 # non basic solution, ie no crossover
barrier.convergetol: 1.e-5
feasopt.tolerance: 1.e-6
cbc-default: {} # Used in CI
glpk-default: {} # Used in CI
mem: 30000 #memory in MB; 20 GB enough for 50+B+I+H2; 100 GB for 181+B+I+H2
walltime: "12:00:00"
# docs :
custom_files:
activate: false
files_path: ''
network_name: ''
Hey @ricnogfer! So the issue with simplify_network can be solved by setting the weather year to 2018. Currently, residential and commercial load profiles are tied to 2018 weather data, so thats required. You can updated your config to these settings. Sorry, there should be an assertion for this in the code.
# docs :
scenario:
interconnect: [western] #"usa|texas|western|eastern"
clusters: [33]
simpl: [75]
opts: [REM-336h]
ll: [v1.0]
sector: "G" # G
planning_horizons: [2030] #(2018-2023, 2030, 2040, 2050)
foresight: 'perfect' # myopic, perfect
model_topology:
transmission_network: 'reeds' # [reeds, tamu]
topological_boundaries: 'reeds_zone' # [county, reeds_zone]
interface_transmission_limits: false
include: # mixed zone types not supported
# reeds_zone: []
# reeds_state: ['CA']
# reeds_ba: []
aggregate: # eligible keys: [reeds_zone, trans_reg]
# trans_grp: []
# reeds_zone: []
# docs :
enable:
build_cutout: false
renewable_weather_years: [2018]
snapshots:
start: "2018-01-01"
end: "2019-01-01"
inclusive: "left"
I do get a very similar error to your original error now, but at solve_network, which is kinda odd. Just looking into that now.
INFO:gurobipy:
INFO:linopy.constants: Optimization successful:
Status: ok
Termination condition: optimal
Solution: 137488 primals, 542412 duals
Objective: 7.65e+10
Solver model: available
Solver message: 2
INFO:pypsa.optimization.optimize:The shadow-prices of the constraints Link-ashp_cooling_generation, Link-gshp_cooling_generation, land_use_constraint were not assigned to the network.
Traceback (most recent call last):
File "/local-scratch/localhome/tmb8/repos/pypsa-usa/workflow/.snakemake/scripts/tmpcwbwi_lz.solve_network.py", line 1912, in <module>
n.export_to_netcdf(snakemake.output[0])
File "/local-scratch/localhome/tmb8/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pypsa/io.py", line 816, in export_to_netcdf
_export_to_exporter(
File "/local-scratch/localhome/tmb8/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pypsa/io.py", line 583, in _export_to_exporter
col_export = pnl[attr].columns
^^^^^^^^^^^^^^^^^
File "/local-scratch/localhome/tmb8/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/generic.py", line 6299, in __getattr__
return object.__getattribute__(self, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Series' object has no attribute 'columns'
Thanks for the feedback @trevorb1!
Unfortunately, setting the weather year to 2018, as suggested, didn't solve the issue. Do you have any idea on what is going on? Many thanks in advance.
You changed the snapshots as well? Sorry, should have mentioned that in the original comment. I havent sorted out the issue in solve network, yet. Just trying to trace back to where this issue popped up.
Yes, I did change the snapshots as well in the config YAML file. Here it is:
renewable_weather_years: [2018]
snapshots:
start: "2018-01-01"
end: "2019-01-01"
inclusive: "left"
Unfortunately, these changes still do not solve the issue. Thanks for looking into it @trevorb1 !
Hmm... your running on main or develop? I doing my testing off of main right now.
Hey @ricnogfer, sorry, just following up on this. This error persists for you in simplify_network still, or you get to solve_network?
On the solve_network, the issue has to do with calculating the mu value for the GlobalConstraint on the carbon limits. I'm still trying to resolve exactly whats happening. But I have not been able to replicate your simplify_network error.
Hi @trevorb1, thanks for looking into this!
On my side, it's failing on the solve_network logic right after the optimizer (Gurobi in this case) has solved the network. Here is the (error) output log:
INFO:gurobipy:
Barrier solved model in 92 iterations and 46.24 seconds (50.21 work units)
INFO:gurobipy:Barrier solved model in 92 iterations and 46.24 seconds (50.21 work units)
Optimal objective 7.21538796e+10
INFO:gurobipy:Optimal objective 7.21538796e+10
INFO:gurobipy:
INFO:linopy.constants: Optimization successful:
Status: ok
Termination condition: optimal
Solution: 84061 primals, 326185 duals
Objective: 7.22e+10
Solver model: available
Solver message: 2
INFO:pypsa.optimization.optimize:The shadow-prices of the constraints Link-ashp_cooling_generation, Link-gshp_cooling_generation, land_use_constraint were not assigned to the network.
Traceback (most recent call last):
File "/home/rick/workspace_usa2/pypsa-usa/workflow/.snakemake/scripts/tmp_kswtwn6.solve_network.py", line 1912, in <module>
n.export_to_netcdf(snakemake.output[0])
File "/home/rick/workspace_usa2/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pypsa/io.py", line 816, in export_to_netcdf
_export_to_exporter(
File "/home/rick/workspace_usa2/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pypsa/io.py", line 583, in _export_to_exporter
col_export = pnl[attr].columns
^^^^^^^^^^^^^^^^^
File "/home/rickworkspace_usa2/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/generic.py", line 6299, in __getattr__
return object.__getattribute__(self, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Series' object has no attribute 'columns'
[Tue May 27 00:17:27 2025]
Error in rule solve_network:
jobid: 2
input: resources/western/elec_s75_c33_ec_lv1.0_REM-580h_E-G.nc, repo_data/ReEDS_Constraints/transmission/transmission_capacity_init_AC_ba_NARIS2024.csv, config/policy_constraints/reeds/prm_annual.csv, config/policy_constraints/reeds/rps_fraction.csv, config/policy_constraints/reeds/ces_fraction.csv, resources/western/pop_layout_elec_s75_c33.csv, config/policy_constraints/ev_policy.csv
output: results/demoF/western/networks/elec_s75_c33_ec_lv1.0_REM-580h_E-G.nc, results/demoF/western/configs/config.elec_s75_c33_lv1.0_REM-580h_E-G.yaml
log: logs/demoF/solve_network/western/elec_s75_c33_ec_lv1.0_REM-580h_E-G_solver.log, logs/demoF/solve_network/western/elec_s75_c33_ec_lv1.0_REM-580h_E-G_python.log (check log file(s) for error details)
conda-env: /home/rick/workspace_usa2/pypsa-usa/workflow/.snakemake/conda/f06d420971abea4d3417b140141e32d3_
RuleException:
CalledProcessError in file /home/rick/workspace_usa2/pypsa-usa/workflow/rules/solve_electricity.smk, line 60:
Command 'set -euo pipefail; /home/rick/workspace_usa2/miniforge3/envs/pypsa-usa/bin/python3.11 /home/rick/workspace_usa2/pypsa-usa/workflow/.snakemake/scripts/tmp_kswtwn6.solve_network.py' returned non-zero exit status 1.
File "/home/rick/workspace_usa2/pypsa-usa/workflow/rules/solve_electricity.smk", line 60, in __rule_solve_network
File "/home/rick/workspace_usa2/miniforge3/envs/pypsa-usa/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2025-05-27T000418.448426.snakemake.log