siphon icon indicating copy to clipboard operation
siphon copied to clipboard

download() method can blow through memory

Open dopplershift opened this issue 4 years ago • 1 comments

Try to include:

from siphon.catalog import TDSCatalog
cat = TDSCatalog('http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p25deg/catalog.xml')
cat.latest.download()  # Don't do this unless you want to use all memory

The problem is that the implementation of download() is pretty naive:

        if filename is None:
            filename = self.name
        with self.remote_open() as infile:
            with open(filename, 'wb') as outfile:
                outfile.write(infile.read())

which means for the 0.25 GFS, it will download an entire 20-30GB run, and only then write it to disk. Not sure if a solution to #139 would fix this, but it might be better to just immediately fix this by reading and writing in e.g 1M block sizes.

dopplershift avatar Oct 10 '19 03:10 dopplershift

shutil.copyfileobj or one of its friends may be helpful here.

dopplershift avatar Oct 21 '19 01:10 dopplershift