gpt-crawler icon indicating copy to clipboard operation
gpt-crawler copied to clipboard

Published 20 hours ago • verified publisher icon BuilderIO

Crawling duplicated url

Open hbakhtiyor opened this issue 7 months ago • 1 comments

need to ignore the url if already crawled the page, in my case same url crawled several times

hbakhtiyor avatar Nov 18 '23 10:11 hbakhtiyor

请问怎么控制相同的网址是否可以抓取多次，我需要每次抓取都去实时的抓取

18758976569 avatar Feb 06 '24 08:02 18758976569