rosepearson
performance improvement - separate out raster generation for each waterway
Currently we produce a single waterway raster for all waterways considered together. This leads to errors with large files. consider generating each separately as generally these will be much smaller than the overall waterway network. This should simplify memory management and reduce errors with files failing to be saved.
Actions:
- Save each waterway separately
- Save each stopbank separately
Consider defining a search width separate to the feature width to ensure more valid returns.
Example showing where no returns at some bits of the river:
Note failing CI seems related to issues specific to netcdf4 1.7.1 on https://github.com/Unidata/netcdf4-python/issues?q=is%3Aissue%20state%3Aopen%20hdf%20error
This si something @jennan ran into recently as referenced in this commit - https://git.niwa.local/forecasting-services/packages/highres_nwm/-/commit/f45845f7a19739f73c94d920e6acd86301d60020 he migrated to h5netcdf and also builds his environments using pip.
Further notes on CI errors related to saving / loading rasters:
Comparing the last successful CI tests: https://github.com/rosepearson/GeoFabrics/actions/runs/10825077248/job/30033393178 against the first failing tests: https://github.com/rosepearson/GeoFabrics/actions/runs/10839705758/job/31267478081
Library versions (All conda installed): netcdf4 - 1.7.1 - same libnetcdf - 4.9.2 - same libgdal-hdf5 - 3.9.2 - same libgdal-netcdf - 3.9.2 - same libgdal-hdf4 - 3.9.2 - same hdf4 - 4.2.15 - same hdf4 - 1.14.3 - same rioxarray - 0.17.0 same rasterio - 1.3.11 - same