getting OISST

[mdsumner]

I'm having this bomb out:

 downloading file 2646 of 15503: https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198811/oisst-avhrr-v02r01.19881128.nc ...
 bb_rget exited with an error (OpenSSL SSL_connect: Connection reset by peer in connection to www.ncei.noaa.gov:443 )

I tried it again, to see if it was similar but it bombed earlier:

 file unchanged on server, not downloading.
 downloading file 319 of 15504: https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198207/oisst-avhrr-v02r01.19820716.nc ...
  |                                                                      |   0%
 file unchanged on server, not downloading.
 downloading file 320 of 15504: https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198207/oisst-avhrr-v02r01.19820717.nc ...
 bb_rget exited with an error (Timeout was reached: [www.ncei.noaa.gov] SSL connection timeout)

Tue Feb 13 04:18:50 2024 dataset synchronization complete: NOAA OI 1/4 Degree Daily SST AVHRR
>
>
>
> proc.time()
    user   system  elapsed
  71.713    2.780 1075.225

trying again with 'wait = 10'

[raymondben]

wait will probably help, but at a guess it'll still happen. It looks like it might be intermittent network flakiness that eventually manifests itself because there are so many files. Maybe need to add a retries parm ...

[mdsumner]

ok that's helpful, I actually ran the whole read every file from URL, and only a handful of fails - but not sure if they were my out of date list (with preliminary ones) or the network.

the cool thing with slurm + parallel mcmapply is it automatically has a try/catch in it so you get most of the results

is there any problem with bowerbird just giving up on a file rather than failing ?

[mdsumner]

it's still going with wait = 10, seems ok!!