icrawler icon indicating copy to clipboard operation
icrawler copied to clipboard

Published 20 hours ago • verified publisher icon hellock

Duplicate URLs?

Open MollyDream opened this issue 11 months ago • 1 comments

Hello! I was reading the implementation and got a bit surprised in finding that there seems to be no check for duplicate URLs before adding them to the queue. Is it not needed naturally, or am I missing something? Thank you so much!

MollyDream avatar Dec 28 '24 10:12 MollyDream

Yeah I think that would be a nice feature to have. Would you mind to submit a PR for it?

ZhiyuanChen avatar Jan 02 '25 06:01 ZhiyuanChen