aw-server-rust
aw-server-rust copied to clipboard
Worker thread crashes (all API requests timeout) after disk has been full
My system drive got full the other day, and today I noticed aw-server-rust wasn't responding to API requests (but was happily serving the web UI).
After a while I realized what may have caused it, and indeed found in the logs:
thread '<unnamed>' panicked at 'Failed to commit datastore transaction! database or disk is full', aw-datastore/src/worker.rs:182:29
So, when the disk gets full, the DB worker thread panics, and future DB requests will just fail to respond (since the worker is dead).
I've seen this a few times before as well, have forgotten to document it.
What do you think the correct behaviour should be? The options i see are:
- Shut down aw-server-rust completely so the user gets notified by aw-qt
- Drop all transactions until the disk is no longer full, but print warnings in the log.
I think the server should respond with HTTP code 503, with a suitable error in the log. That way, watchers can queue events in memory, as appropriate.
On Mon, Nov 15, 2021 at 10:54 PM Johan Bjäreholt @.***> wrote:
I've seen this a few times before as well, have forgotten to document it.
What do you think the correct behaviour should be? The options i see are:
- Shut down aw-server-rust completely so the user gets notified by aw-qt
- Drop all transactions until the disk is no longer full, but print warnings in the log.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ActivityWatch/aw-server-rust/issues/256#issuecomment-969357232, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKXDOTMBMVHWTQZ6E6K6XDUMF6RNANCNFSM5ICMDH7A .