Accessing xarray when using goes_nearesttime loaded in memory

[nrchade]

hour = '2022-01-01 00:00:00'
g = goes_nearesttime(hour,
                   satellite='goes16',
                   product='ABI',
                   domain = 'C',
                   return_as='xarray',
                   download = False)

timestamp = str(g.time_coverage_end.values)
R = g['CMI_C02'].data

results in this error: raise ValueError("I/O operation on closed file.") ValueError: I/O operation on closed file.

R = g['CMI_C02'].data

When download = True and return as filelist, I have no issues opening xarray.open_dataset, but these files are huge so i was hoping to load in memory, process, and then move onto the next one. Am i doing something wrong when trying to access the data once this is loaded into memory?

[blaylockbk]

Hi @nrchade,

Thanks for reporting this behavior. I admit, the in-memory processing is not well executed in goes2go (would love to see a pull request improving the behavior).

I haven't worked on or tested the download=False option much because in my initial experimentation, it took longer to load the data directly into memory and process the data than downloading and opening the file. I think this is because when xarray opens a NetCDF file on disk, it doesn't necessarily read the full file into memory, just the parts you ask for (if that makes sense). If that is still true (that working with a downloaded file is faster than loading it into memory), then I'd recommend downloading the file and deleting it after you do your processing.

I'll leave this issue open and try to find time to investigate the I/O operation on closed file issue. Just so you are aware, goes2go is a hobby project, so I can't promise I'll get to it too soon, but I'll try.

[nrchade]

Hi @blaylockbk,

I ultimately did what you suggested, download the file, process, then deleted it to move to the next one. It did seem faster to load into memory when it did download which seemed odd.

I think your work on this package is to be commended as you have helped to make working with this type of data easier for the layperson. I ultimately needed a small amount of info specific to a certain geography and prior attempts to solving this problem seemed overwhelming.

Many thanks!

[blaylockbk]

Adding this for reference

To load data directly into memory, need to add "mode=bytes" to the URL path

[nrchade]

Outstanding!