grab-site icon indicating copy to clipboard operation
grab-site copied to clipboard

Published 20 hours ago • verified publisher icon ArchiveTeam

Pause gracefully if OSError (No space left on device)

Open TheTechRobo opened this issue 4 years ago • 4 comments

Instead of crashing the crawl, why don't we simply pause and retry every so often if the computer has no space left?

TheTechRobo avatar Oct 30 '21 16:10 TheTechRobo

I believe wpull and ludios_wpull do this if psutil is installed (psutil>=2.0,<=4.2 is in its requirements). I think I didn't include the requirement in grab-site for compatibility with Windows or macOS? Not sure.

https://github.com/ArchiveTeam/grab-site/blob/6269289a2ca874bae52f116016ca54dc8887d0cc/extra_docs/pause_resume_grab_sites.sh is the workaround I use to avoid crashes right now.

I agree this should be better and work by default.

ivan avatar Oct 31 '21 02:10 ivan

Well, a 50GB crawl I had crashed. :/

Just installed it for the future.

TheTechRobo avatar Oct 31 '21 03:10 TheTechRobo

Nope, didn't do anything. I have psutil yet my 33GB crawl just crashed because I ran out of space.

TheTechRobo avatar Nov 02 '21 20:11 TheTechRobo

Where does this error pop up, I can't remember but I want to fix this.

Do you think just putting it in a try block would work and catch OSErrors, and pause the crawl, or should we do something else?

TheTechRobo avatar Jan 02 '22 14:01 TheTechRobo