lun-4
database maintenance ideas
this idea is being shamelessly stolen from hydrus, basically, there are very good maintenance tasks to be done across the index file
- making sure hashes are consistent across the entire store
- running
VACUUM; COMMITfrom time to time- maybe we could make a full breaking change requiring dumping and reimporting the database by enabling incremental auto vaccuuming? (sqlite docs say you must do that before creating any tables)
our problem, compared to hydrus, is that we are a suite of little tools that interact with the index. hydrus automatically decides when to do maintenance from time to time (say, on some program exits, or letting it run in the background in low power mode), but how can we do that when the only current tool that is long lived is awtfdb-watcher?
maybe we need some kind of job system? or would a awtfdb-janitor program that's plugged to cron.daily be enough? we also need to think about communicating incorrect hashes, because if we let this run at a time the user is asleep, emitting a desktop notification won't do shit for them. maybe some kind of notification queue inside the database, and as every tool loads the database up, they emit the notifications themselves? that sounds better.
https://github.com/lun-4/awtfdb/commit/a37d8b89b2ec0767ed0bda0c4016a0ace84a95bc adds awtfdb-janitor which validates database consistency against the filesystem, both file existence and file content hashes (https://github.com/lun-4/awtfdb/commit/222226b5f523f63744c87f65a1243df294eb01f2) are verified.
TODO:
- vacuuming?
- notification strategy for checks that find issues
- garbage collect hash entries: https://github.com/lun-4/awtfdb/commit/6fff84d6b562a980c6b8ab04523d9391aa4585e5
- validate tag core consistency: https://github.com/lun-4/awtfdb/commit/0c2ca263a906ae8e5a8cc175f1579f0367b72e77
VACUUMing is a very heavy operation that should be left to manual work.
notification strategy is left for the user to do if they plug awtfdb-janitor into a cron task. we do return 1 exit code when we find errors, so yeah.
leaving this task as done