Herbie icon indicating copy to clipboard operation
Herbie copied to clipboard

On Windows, Herbie can't remove subset file

Open blaylockbk opened this issue 3 years ago • 4 comments

When I open a subset from with xarray, Herbie tries to remove the file if it didn't exist before (some basic clean up). But this doesn't work on Windows

H = Herbie(
    "2021-10-9",
    model="hrrr",
    product="prs",
)

ds = H.xarray('^TMP:2 m')
PermissionError: [WinError 32] 
The process cannot access the file because it is being used by another process: 
'C:\\Users\\blayl_depgywe\\data\\hrrr\\20211009\\hrrr.t00z.wrfprsf00.grib2.subset_8746b7e5d534efa196e92e53c61ec747f4c936a5'

The offending line is Line 761: local_file.unlink() # Removes file

blaylockbk avatar Oct 09 '21 14:10 blaylockbk

Hmmm, the cfgrib.open_datasets doesn't make it easy to close the file.

blaylockbk avatar Oct 09 '21 15:10 blaylockbk

It was also reported that the init.py is not creating the config file.

So, is this a Windows issue or a Python 3.9 on Windows issue?

blaylockbk avatar Nov 18 '21 16:11 blaylockbk

Perhaps all I need is a os.close(local_file) statement

blaylockbk avatar Feb 17 '22 04:02 blaylockbk

Agh, still trying to figure this out.

https://docs.python.org/3/library/os.html#os.remove On Windows, attempting to remove a file that is in use causes an exception to be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use.

blaylockbk avatar Feb 17 '22 05:02 blaylockbk

I might have found a solution...use xr.load_dataset instead. This loads the data into memory and closes the file.

https://docs.xarray.dev/en/stable/generated/xarray.load_dataset.html

blaylockbk avatar Jan 06 '23 22:01 blaylockbk

Oh wow. How did I miss this one...I simply forgot to close the file opened by pygrib. Well, 2785b61 seems to have fixed this one. I'll do a bit more testing, but I'm certain this is resolved.

blaylockbk avatar Jan 17 '23 04:01 blaylockbk