apache
CouchDB dies if log directory is full
CouchDB dies and systemd tries to restart it over and over, if the log directory for file based logging is full.
Steps to Reproduce
Fill the disk (for example with fallocate) where the CouchDB log is written to and watch CouchDB go down.
Expected Behaviour
I expect, that CouchDB just stops logging and still works. The CouchDB data directory is on a separate disk and still has plenty of space to work with.
Your Environment
Debian 12 with the official CouchDB 3.3.3 package.
Good idea to stop writting and perhaps bump a metric.
Another way to handle it might be to use rsyslog or log to journald vs just a plain file.
We switched to using journald for RPM-based distros in 3.4.0, for instance.
The very first thing I learned when administering a unix-like was to never let a fs fill. Bad things happen! Pretty straight forward stuff.
Is not the solution here proper log management?
@mikkimichaelis good point. But I think CouchDB's simple file logger is just that - a basic logger. I think it would be good to enhance it to stop writing when the disk is full, however I don't know if it will gain proper file rotation, it's own remote logging, or other such features.
In production it might be better to use a proper logger like rsyslog or journald [1]. CouchDB can be configured to use those easily:
[1] https://docs.couchdb.org/en/stable/config/logging.html#log/syslog_host