fbcrawl icon indicating copy to clipboard operation
fbcrawl copied to clipboard

Published 20 hours ago • verified publisher icon rugantio

csv file limit

Open gordon4067 opened this issue 5 years ago • 1 comments

Hi, I'm crawling the comments of one of famous guy's post.

Some comments are up to 50000+ and I found that the scrapy engine would stop to crawler when it crawls at about 10000 comments. I checked my fake fb account and it was not been banned. Is there any limit at the csv file? If it is, it there any method to solve it.

Really appreciate about this tool.

gordon4067 avatar May 26 '19 16:05 gordon4067

yes and you can change it:

go to line 75 in fbcrawl.py , increase 10e5 to 10e6 (a million) or whatever you want...

Raki22 avatar Jan 24 '20 23:01 Raki22