pictrs orphaned temp files

[pixlguru]

Issue Summary

in pictrs containers' /tmp/ directory orphaned magick-* temporary files are accumulating, (like 10GB in 2 weeks)

Technical details

errors in pictrs logs:

Sep 20 17:47:28.407  INFO actix_server::builder: SIGTERM received, stopping    
Sep 20 17:47:28.407  INFO actix_server::worker: Shutting down worker, 0 connections    
Sep 20 17:47:28.408  INFO actix_server::worker: Shutting down worker, 0 connections    
Sep 20 17:48:15.259 ERROR sled::config: cache capacity is limited to the cgroup memory limit: 209715200 bytes    
Sep 20 17:48:15.340 ERROR sled::config: cache capacity is limited to the cgroup memory limit: 209715200 bytes    
Sep 20 17:48:15.356  INFO actix_server::builder: Starting 2 workers    
Sep 20 17:48:15.356  INFO actix_server::builder: Starting "actix-web-service-0.0.0.0:8080" service on 0.0.0.0:8080```

[dessalines]

cc @asonix

[asonix]

The logs presented are not related to the problem, and are actually not important. Sled doesn't like when you constrain it's RAM but it won't cause issues.

Cleaning up ImageMagick's tmp files happens automatically in pictrs 0.3, but I'm not sure how I'd get it working in 0.2. I can start looking into this, though

[dessalines]

Its pry not necessary, I'd rather focus on moving to 0.3 when you're ready for that.

[pixlguru]

ok, as a workaround i just run scheduled docker exec -it lemmy_pictrs_1 /bin/bash -c "rm /tmp/magick-*"

[dessalines]

In that case I'll close this, but of course this'll be searchable.

[pixlguru]

The issue still persists, it recently crashed the server

[Nutomic]

Which pictrs version are you using? And how did temp files crash your server? Do you have a log?

[pixlguru]

I'm using the version the ansible setup provides. The server crashed cause it ran out of disk space, corrupting the database. It was caused by leftover pictrs magick-* temporary files inside it 's docker containers /tmp folder

[pixlguru]

After digging some more into it, I found these entries in syslog syslog

[pixlguru]

I'll try increasing pictrs mem_limit and see how this works out

[asonix]

that does point to an issue in pict-rs for cleaning up tmp files when imagemagick, ffmpeg, or exiftool get killed. I'll look into this for 0.4 stable

[asonix]

oh, just re-read your command for cleaning the orphaned files. those magick- files are left behind by imagemagick, and i'm hesitant to add logic to pict-rs to clean them in case they're still in use. I think increasing the container's memory limit is probably the correct solution here

[asonix]

alternatively, you could play with the values in imagemagick's policy xml file. This is the default one pict-rs ships with: https://git.asonix.dog/asonix/pict-rs/src/branch/main/docker/object-storage/root/usr/lib/ImageMagick-7.1.0/config-Q16HDRI/policy.xml

it needs to be mounted to /usr/lib/ImageMagick-7.1.0/config-Q16HDRI/policy.xml inside the container

[pixlguru]

Seems the files are left behind if pictrs crashes due to OOM. Increasing the containers mem_limit resolved the issue