bashpider
bashpider copied to clipboard
Xarg & Wget stops working after about an hour
Running your script with Cygwin on Windows XP with Dual Core and 4GB Ram
I run the script with ./crawl.sh urlfile.txt 50
I am trying to trawl through 4GB of URL
The Bash shell seems to stop after about an hour and the downloads stop even though its only 2% through the URL list
Any ideas at what could be wrong?
Many Thanks