pyschism icon indicating copy to clipboard operation
pyschism copied to clipboard

Cached NWM data still requires internet access

Open SorooshMani-NOAA opened this issue 1 year ago • 1 comments

Using ensembleperturbation we setup many SCHISM run directories. Since this can be compute intensive for large ensembles we run the ensemble generation on compute nodes (no internet). In order to avoid the issue of downloading the data from compute node, I first setup a single run to cache the data and then setup the rest on the compute node.

At first I thought this solution should work, but then I realized PySCHISM still needs to have internet because the caching kicks in for data, but still metadata is fetched from the internet every time. See: Metadata https://github.com/schism-dev/pyschism/blob/fc41b51cc751d9b64a75c91fac48e6a7f0a27e93/pyschism/forcing/source_sink/nwm.py#L490-L498 and Data https://github.com/schism-dev/pyschism/blob/fc41b51cc751d9b64a75c91fac48e6a7f0a27e93/pyschism/forcing/source_sink/nwm.py#L524-L525

SorooshMani-NOAA avatar Apr 08 '24 20:04 SorooshMani-NOAA

As a short term solution I can just copy paste the downloaded data from one setup directory to another, but since I'm using automation, around pyschism I was hoping to use the built-in caching capability in this case.

SorooshMani-NOAA avatar Apr 08 '24 20:04 SorooshMani-NOAA