pyaerocom icon indicating copy to clipboard operation
pyaerocom copied to clipboard

Mandatory data_revision on all obs-readers, disable last-file check in caching

Open heikoklein opened this issue 7 months ago • 0 comments

Is your feature request related to a problem? Please describe. Caching of ungridded data relies on both data_revision and newest_file_in_read_dir, see #1242 . The newest_file_in_read_dir test is error prone since a touch or another minor filesystem operation will invalidate the cache, in addition it can be very slow, e.g. for the EEA-reader. If observation-readers deliver their own mandatory data_revision, the too general test can be removed.

Describe the solution you would like to see

  • [ ] All observation readers must have a data_revision. This is already addressed in pyaro in https://github.com/metno/pyaro/issues/37, the readers in pyaerocom must be checked
  • [ ] newst_file_in_read_dir and newest_file_date_in_read_dir are removed from the ungridded-cache as well as the test-code: https://github.com/metno/pyaerocom/blob/300de5a33a601a0ce2b467104e71dc00e8b20228/pyaerocom/io/cachehandler_ungridded.py#L163

heikoklein avatar Jul 05 '24 10:07 heikoklein