elastic-ci-stack-for-aws icon indicating copy to clipboard operation
elastic-ci-stack-for-aws copied to clipboard

Provide a way of customizing `DOCKER_PRUNE_UNTIL`, `DISK_MIN_AVAILABLE`, and `DISK_MIN_INODES` for the cron job

Open arturopie opened this issue 6 years ago • 4 comments

Right now, there is no way of customizing those values for the hourly docker-gc cron job. The defaults are not good enough for us since our docker images are quite big and the disk gets fill up very quick. We also run a RAM drive for the runners, which requires different clean up params from the builders.

I imagine and approach could be to add those variables to the cloud formation template parameters, and generate the cron job on provision time, rather than image building time. Another option is to generate a file on provision time that the cron job would source when it runs. Thoughts?

arturopie avatar Aug 03 '18 19:08 arturopie

Yeah, I think this would be worth adding a stack parameter for.

lox avatar Oct 01 '18 01:10 lox

+1 on this, are there any plans on adding this?

tspecht avatar Jan 12 '19 13:01 tspecht

This would definitely be nice to have. Would happily accept a PR.

lox avatar Jul 27 '19 23:07 lox

We have builds start failing when there's still a significant amount of disk free (~2GB), because our images are large; a large chunk of these images can be automatically cleaned with the docker system prune (even with the until=1h filter), except the DISK_MIN_AVAILABLE threshold is too low, so the script doesn't end up running the prune.

With the current 4.3 stacks, the limit seems to be 1GB, but on master it seems to be 5GB (https://github.com/buildkite/elastic-ci-stack-for-aws/commit/e97921cc74e8a69fccbe98ebcc6ee0f50a9ccbcf). That would be good-enough as a work-around for us. Is there a reason it isn't in the 4.3 series?

huonw avatar Nov 20 '19 05:11 huonw