pypsa-usa icon indicating copy to clipboard operation
pypsa-usa copied to clipboard
verified publisher icon PyPSA

Error using `uv` with the `build_renewable_profiles` rule

Open trevorb1 opened this issue 10 months ago • 1 comments

Checklist

  • [x] I am using the current master branch
  • [x] I am running on an up-to-date pypsa-usa environment. Update via conda env update -f envs/environment.yaml

The Issue

If I run though conda, the workflow works fine. If I run through uv, I get an error in the build_renewable_profiles rule.

Steps To Reproduce

  1. Install UV environment
  2. Change weater profile to 2018
  3. Run uv run snakemake -j1 --scheduler-ilp-solver GUROBI_CMD

Expected Behavior

No response

Error Message

[Sat Feb 15 18:38:20 2025]
rule build_renewable_profiles:
    input: data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_USA_EPSG-4326.tif, data/natura.tiff, data/gebco/gebco_2023_n55.0_s10.0_w-126.0_e-65.0.tif, resources/Western/western/Geospatial/country_shapes.geojson, resources/Western/western/Geospatial/offshore_shapes.geojson, repo_data/geospatial/CEC_GIS/CEC_Wind_BaseScreen_epsg3310.tif, repo_data/geospatial/CEC_GIS/CEC_Solar_BaseScreen_epsg3310.tif, repo_data/geospatial/boem_osw_planning_areas.tif, resources/Western/western/Geospatial/regions_offshore.geojson, cutouts/usa_era5_2019.nc
    output: resources/Western/western/profile_offwind_floating.nc, results/Western/western/land_use_availability_offwind_floating.png
    log: logs/Western/western/build_renewable_profile_offwind_floating.log
    jobid: 18
    benchmark: benchmarks/Western/western/build_renewable_profiles_offwind_floating
    reason: Missing output files: resources/Western/western/profile_offwind_floating.nc; Input files updated by another job: cutouts/usa_era5_2019.nc
    wildcards: interconnect=western, technology=offwind_floating
    threads: 6
    resources: tmpdir=/tmp, mem_mb=103794, mem_mib=98986

INFO:__main__:using cutout "cutouts/usa_era5_2019.nc"
INFO:__main__:Calculate landuse availability...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 666, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 361, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 286, in shape_availability
    excluder.open_files()
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 497, in open_files
    geometry = geometry.to_crs(self.crs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/geoseries.py", line 1207, in to_crs
    self.values.to_crs(crs=crs, epsg=epsg), index=self.index, name=self.name
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/_compat.py", line 90, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/array.py", line 1019, in to_crs
    new_data = transform(self._data, transformer.transform)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/array.py", line 1750, in transform
    result[~has_z] = shapely.set_coordinates(
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/shapely/coordinates.py", line 195, in set_coordinates
    lib.set_coordinates(geometry_arr, coordinates)
shapely.errors.GEOSException: IllegalArgumentException: Points of LinearRing do not form a closed linestring
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmph9eszlla.build_renewable_profiles.py", line 176, in <module>
    availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 747, in compute_availabilitymatrix
    availability = list(pool.map(_process_func, shapes.index))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
      ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 666, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 361, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
      ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 286, in shape_availability
    excluder.open_files()
  ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/atlite/gis.py", line 497, in open_files
    geometry = geometry.to_crs(self.crs)
^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/geoseries.py", line 1207, in to_crs
    self.values.to_crs(crs=crs, epsg=epsg), index=self.index, name=self.name
^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/_compat.py", line 90, in wrapper
    return func(*args, **kwargs)
  ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/array.py", line 1019, in to_crs
    new_data = transform(self._data, transformer.transform)
  ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/geopandas/array.py", line 1750, in transform
    result[~has_z] = shapely.set_coordinates(
      ^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/.venv/lib/python3.11/site-packages/shapely/coordinates.py", line 195, in set_coordinates
    lib.set_coordinates(geometry_arr, coordinates)
      ^^^^^^^^^^^^^^^^^
shapely.errors.GEOSException: IllegalArgumentException: Points of LinearRing do not form a closed linestring
[Sat Feb 15 18:38:29 2025]
Error in rule build_renewable_profiles:
    jobid: 18
    input: data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_USA_EPSG-4326.tif, data/natura.tiff, data/gebco/gebco_2023_n55.0_s10.0_w-126.0_e-65.0.tif, resources/Western/western/Geospatial/country_shapes.geojson, resources/Western/western/Geospatial/offshore_shapes.geojson, repo_data/geospatial/CEC_GIS/CEC_Wind_BaseScreen_epsg3310.tif, repo_data/geospatial/CEC_GIS/CEC_Solar_BaseScreen_epsg3310.tif, repo_data/geospatial/boem_osw_planning_areas.tif, resources/Western/western/Geospatial/regions_offshore.geojson, cutouts/usa_era5_2019.nc
    output: resources/Western/western/profile_offwind_floating.nc, results/Western/western/land_use_availability_offwind_floating.png
    log: logs/Western/western/build_renewable_profile_offwind_floating.log (check log file(s) for error details)

RuleException:
CalledProcessError in file /home/trevor/master/pypsa-usa/workflow/rules/build_electricity.smk, line 199:
Command 'set -euo pipefail;  /home/trevor/master/pypsa-usa/.venv/bin/python3 /home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmph9eszlla.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/home/trevor/master/pypsa-usa/workflow/rules/build_electricity.smk", line 199, in __rule_build_renewable_profiles
  File "/home/trevor/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

Anything else?

No response

trevorb1 avatar Feb 16 '25 02:02 trevorb1

So I found that if I run the workflow using uv from inside a session that has the conda environment activated, I get this error. But if I run it from a session where I am in my base conda env, I do not get this error. This isn't a solution, I guess since we should add a dependency to the lock file.... will look further.

ktehranchi avatar Mar 05 '25 17:03 ktehranchi

I'm closing this since it's not an issue inherently with our project. We just shouldn't run uv from within a pypsa-usa mamba environment

ktehranchi avatar Aug 19 '25 21:08 ktehranchi