[Feat] Add automatic retries to failed links on crawl

[calebpeffer]

I've had a few customers who are concerned with the few failures they experience during long crawl jobs. Some have even implemented their own automatic retry functionality.

If we handle this automatically, we improve reliability and prevent customers from having to implement this themselves.