siphon
siphon copied to clipboard
download() method can blow through memory
Try to include:
- Code Sample, a copy-pastable example if possible
- A "Minimal, Complete and Verifiable Example" will make it much easier for maintainers to help you
from siphon.catalog import TDSCatalog
cat = TDSCatalog('http://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p25deg/catalog.xml')
cat.latest.download() # Don't do this unless you want to use all memory
The problem is that the implementation of download()
is pretty naive:
if filename is None:
filename = self.name
with self.remote_open() as infile:
with open(filename, 'wb') as outfile:
outfile.write(infile.read())
which means for the 0.25 GFS, it will download an entire 20-30GB run, and only then write it to disk. Not sure if a solution to #139 would fix this, but it might be better to just immediately fix this by reading and writing in e.g 1M block sizes.
shutil.copyfileobj
or one of its friends may be helpful here.