pyschism
pyschism copied to clipboard
Cached NWM data still requires internet access
Using ensembleperturbation we setup many SCHISM run directories. Since this can be compute intensive for large ensembles we run the ensemble generation on compute nodes (no internet). In order to avoid the issue of downloading the data from compute node, I first setup a single run to cache the data and then setup the rest on the compute node.
At first I thought this solution should work, but then I realized PySCHISM still needs to have internet because the caching kicks in for data, but still metadata is fetched from the internet every time. See: Metadata https://github.com/schism-dev/pyschism/blob/fc41b51cc751d9b64a75c91fac48e6a7f0a27e93/pyschism/forcing/source_sink/nwm.py#L490-L498 and Data https://github.com/schism-dev/pyschism/blob/fc41b51cc751d9b64a75c91fac48e6a7f0a27e93/pyschism/forcing/source_sink/nwm.py#L524-L525
As a short term solution I can just copy paste the downloaded data from one setup directory to another, but since I'm using automation, around pyschism I was hoping to use the built-in caching capability in this case.