CroatianMeteorNetwork
Optimize mask, flat, and dark loading with caching
Optimize image calibration loading with caching and improved logging (issue #336 )
- Implement ImageCalibrationLoader class for efficient file caching
- Reduce I/O operations by only reloading files when modified
- Add detailed logging for mask, dark, and flat field image loading
- Fix logging accuracy for all-white mask scenarios (issue #376)
- Implement singleton pattern for consistent cache usage across modules
- Optimize all-white mask check to avoid unnecessary rechecks
This commit should reduces strain on SD cards and hopefully speed up performance on slow storage. Tested on MCL. Need testing on Pi 4 and 3.
This PR just caches the non-bin versions. Should I cache the bins too? Not sure how often the conversions occur, but it wouldn’t be too much work.
Thank you Dave. I flagged the PR as draft for the moment. It probably work fine as is but it could still be improved. I’ll cache the binned versions too. The cache also currently keeps old versions instead of only keeping the most current. Also, there should be a better cleanup process. Sorry for asking to test a bit too early.
It was running fine on Debian, Pi5, 4 last night. I'll check after sunrise, happy to leave those stations there.
I'm testing this version tonight. It caches all binned and non-binned mask, dark, and flat files. The cache should update incrementally if there are any changes to the files. This should reduce I/O operations significantly and decrease CPU load somewhat, as binning is only done when a new file is detected, and checking for all-white masks is only performed when a new mask file is detected - as opposed to every time the files are used (which is continuously).
So far, the only testing I've done is to confirm that it doesn't crash on MCL using mask-only operation and that it loads the correct mask.
What needs to be tested:
- Verification that the cache actually updates when new files are added
- Confirmation that it works with flat and dark files
- Validation that the expected files are loaded correctly
I'm unsure about the actual usage of dark and flat files. I note that dark is disabled by default, and there are currently no options exposed in the config to enable it.
I verified the cache does update when a mask file is updated. Now, if someone could test the flat and dark functions which I have no experience with.