cf-python icon indicating copy to clipboard operation
cf-python copied to clipboard

Published 20 hours ago • verified publisher icon NCAS-CMS

Review file IO model

Open sadielbartholomew opened this issue 4 years ago • 0 comments

We agreed that, during or after the Dask work in #182, we should review the cf approach to file IO.

The topic arose in discussions relating to the new append mode (#30) because files were being inadvertently left open leading to HDF errors and seg faults (e.g. see https://github.com/NCAS-CMS/cf-python/pull/213#issuecomment-847101986) and though this was fixed, we realised there may be improvements to be made regarding the current caching of open files.

Moreover we realised the current file IO model might not work with the parallelisation and lazy loading resulting from Dask-ification so that would probably compel a change in approach, which could be a tweak or could be a large-scale change.

sadielbartholomew avatar May 26 '21 22:05 sadielbartholomew