bundle_config.yaml has incorrect, extra link in bundle_cutouts_northamerica

[mfripp]

Checklist

• [X] I am using the current main branch or the latest release. Please indicate.
• [X] I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

The databundles > bundle_cutouts_northamerica > urls node in configs/bundle_config.yaml specifies both a zenodo URL and a gdrive URL. This bundle should download cutout-2013-era5.nc. The gdrive URL points to that file and could be used to download it successfully. However, the zenodo URL points to an unneeded copy of the worldwide natura.tiff.

The zenodo URL ends up masking the gdrive URL, so that for North American locations, retrieve_databundle_light() creates an unneeded cutouts/natura.tiff and does not create cutouts/cutout-2013-era5.nc. As a result, Snakemake reports that cutouts/cutout-2013-era5.nc is missing after the retrieve_databundle_light rule, and terminates with the error below.

This error can be reproduced by setting countries in config.yaml to include a North American location, e.g., countries: ["DO"], then setting enable: retrieve_databundle: true and running snakemake --cores 1 solve_all_networks.

This error can be fixed by removing the databundles > bundle_cutouts_northamerica > urls > zenodo node from configs/bundle_config.yaml.

Error Message

Bundle successfully loaded and unzipped:
        bundle_landcover_earth
        bundle_cutouts_northamerica
        bundle_natura_earth
        bundle_data_earth
        bundle_hydrobasins
INFO:__main__:Bundle successfully loaded and unzipped:
        bundle_landcover_earth
        bundle_cutouts_northamerica
        bundle_natura_earth
        bundle_data_earth
        bundle_hydrobasins
Waiting at most 5 seconds for missing files.
WARNING:snakemake.logging:Waiting at most 5 seconds for missing files.
MissingOutputException in rule retrieve_databundle_light in file /Users/matthias/Library/CloudStorage/OneDrive-EnvironmentalDefenseFund-edf.org/Research/MCET/pypsa/pypsa-earth/Snakefile, line 149:
Job 9  completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
cutouts/cutout-2013-era5.nc
ERROR:snakemake.logging:MissingOutputException in rule retrieve_databundle_light in file /Users/matthias/Library/CloudStorage/OneDrive-EnvironmentalDefenseFund-edf.org/Research/MCET/pypsa/pypsa-earth/Snakefile, line 149:
Job 9  completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
cutouts/cutout-2013-era5.nc
Removing output files of failed job retrieve_databundle_light since they might be corrupted:
data/ssp2-2.6/2030/era5_2013/SouthAmerica.nc, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/natura/natura.tiff, data/gebco/GEBCO_2021_TID.nc, data/ssp2-2.6/2030/era5_2013/Oceania.nc, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/ssp2-2.6/2030/era5_2013/NorthAmerica.nc, data/ssp2-2.6/2030/era5_2013/Asia.nc, data/ssp2-2.6/2030/era5_2013/Europe.nc, data/eez/eez_v11.gpkg, data/hydrobasins/hybas_world.shp, data/landcover
WARNING:snakemake.logging:Removing output files of failed job retrieve_databundle_light since they might be corrupted:
data/ssp2-2.6/2030/era5_2013/SouthAmerica.nc, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/natura/natura.tiff, data/gebco/GEBCO_2021_TID.nc, data/ssp2-2.6/2030/era5_2013/Oceania.nc, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/ssp2-2.6/2030/era5_2013/NorthAmerica.nc, data/ssp2-2.6/2030/era5_2013/Asia.nc, data/ssp2-2.6/2030/era5_2013/Europe.nc, data/eez/eez_v11.gpkg, data/hydrobasins/hybas_world.shp, data/landcover
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message

[ekatef]

Hello @mfripp! Thanks for spotting.

I confirm that zenodo is used as a priority data source, when extracting data bundle. But in case of North America, the cutout is too big to fit on zenodo drive: that is in fact an hemispherical cutout, due to the fact that it's area intersects the anti-meridian. So, the only possible way to fix the issue now is to remove the zenodo link for NorthAmerica cutout.

If you wish to open a PR to fix that, it will be gratefully accepted!

[ekatef]

The problem should have been fixed with #1055. @mfripp can we close this issue?

[mfripp]

Looks fine to me!

Matthias

From: Ekaterina @.> Date: Thursday, July 18, 2024 at 11:26 AM To: pypsa-meets-earth/pypsa-earth @.> Cc: Matthias Fripp @.>, Mention @.> Subject: [EXTERNAL] Re: [pypsa-meets-earth/pypsa-earth] bundle_config.yaml has incorrect, extra link in bundle_cutouts_northamerica (Issue #1005)

The problem should have been fixed with #1055https://github.com/pypsa-meets-earth/pypsa-earth/pull/1055. @mfripphttps://github.com/mfripp can we close this issue?

— Reply to this email directly, view it on GitHubhttps://github.com/pypsa-meets-earth/pypsa-earth/issues/1005#issuecomment-2236161110, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB4PSFUB4FAWLSVSP6ICNUDZM6J3LAVCNFSM6AAAAABGGJ4DSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZWGE3DCMJRGA. You are receiving this because you were mentioned.Message ID: @.***>

[ekatef]

@mfripp perfect, thanks a lot for the confirmation!