webcrawler
webcrawler copied to clipboard
Feature: implement a retry to go over the links that failed or errors
If the connection is lost, the HTTP requests will fail. There are a few ways to handle this:
- Identify the issue and re-add the link to the queue
- Add a retry option to go over the links that failed or threw an error