The crawler might ask a spider to process an already visited url even if the end_url is already in cache

[Yomguithereal]

We could also register all the urls in the stack but it might get a little costly.