TheTechRobo

Please remind me tomorrow or ill forget :)

Tbh I can't tell. May you tell where you got the github action?

Why not?

Some files will also specify that they should be downloaded in the headers, so that needs to be checked for too.

Well, a 50GB crawl I had crashed. :/ Just installed it for the future.

Nope, didn't do anything. I have psutil yet my 33GB crawl just crashed because I ran out of space.

Where does this error pop up, I can't remember but I want to fix this. Do you think just putting it in a `try` block would work and catch OSErrors,...

Did you end up resolving this issue? If not, large WARC files are typical, even on seemingly small websites. I've had to stop many a crawl because my internet is...

Could this be fixed by killing the wpull connections (documented on the AT Wiki's ArchiveBot page)?

I know, but it'd be a temporary solution.