pyaerocom
pyaerocom copied to clipboard
Mandatory data_revision on all obs-readers, disable last-file check in caching
Is your feature request related to a problem? Please describe.
Caching of ungridded data relies on both data_revision
and newest_file_in_read_dir
, see #1242 . The newest_file_in_read_dir test is error prone since a touch or another minor filesystem operation will invalidate the cache, in addition it can be very slow, e.g. for the EEA-reader. If observation-readers deliver their own mandatory data_revision, the too general test can be removed.
Describe the solution you would like to see
- [ ] All observation readers must have a data_revision. This is already addressed in pyaro in https://github.com/metno/pyaro/issues/37, the readers in pyaerocom must be checked
- [ ]
newst_file_in_read_dir
andnewest_file_date_in_read_dir
are removed from the ungridded-cache as well as the test-code: https://github.com/metno/pyaerocom/blob/300de5a33a601a0ce2b467104e71dc00e8b20228/pyaerocom/io/cachehandler_ungridded.py#L163