civiform icon indicating copy to clipboard operation
civiform copied to clipboard

Detect when Docker is running out of disk space

Open nb1701 opened this issue 1 year ago • 2 comments

When running a local dev instance (run-dev or browser tests), if docker runs out of disk space, the database container will fail to start properly. The failure signature is

PersistenceException: SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms.

But this isn't very clear that the problem is that Docker does not have any more disk space in which to allocate the volume needed for the postgres container.

We should have the script either detect this error condition and print a message saying what the problem probably is. Or, it should detect that Docker is about to run out of available disk space, and offer to do docker image prune for you.

nb1701 avatar Jan 30 '24 19:01 nb1701

The easiest thing is probably going to be to just look for compose exit codes and print a block of text showing sample errors and suggest using prune. We should be able to get some info from docker inspect I believe. The hard part is that getting info from docker on storage and io usage; it is very slow operation.

I wouldn't have it automatically invoke prune and let it sit waiting for user input. It'll jut make the scripts more complicated when taking CI usage into account. Currently there's a lot of indirection on how the docker compose stuff works in our scripts which makes it much harder to figure out what's going on.

Or we don't do anything and teach people how to diagnose issues.

gwendolyngoetz avatar Feb 13 '24 06:02 gwendolyngoetz

We should ensure this works for codespaces as well.

dkatzz avatar Apr 19 '24 21:04 dkatzz