icrawler

icrawler copied to clipboard

Hey! I am trying to download 3000 images for each keyword using BingImageCrawler but I am getting cut off at < 1000 images per keyword. The documentation says To crawl more than 1000 images with a single keyword, we can specify different date ranges. This works easily for GoogleImageCrawler because it supports a tuple of date ranges but BingImageCrawler doesn't (only supports “pastday”, “pastweek”, “pastmonth”, “pastyear”).

How can I set the configs such that I can get more than 1000 images per keyword? Also, if I need to do multiple request for each keyword, how can I set the date filter so that I get non-overlapping images? (Using any two of “pastday”, “pastweek”, “pastmonth”, “pastyear” will always result in overlapping images).

I have the same problem. Please help.

Well, this would require some handcrafts. There are only a few hard-coded cutoffs, and if you remove those, you can crawl more images. However, I do not encourage doing this, as the cutoffs are there for a reason.

Some where i saw that it is possible to search by period. Buy i write the filter and i did not work.

I used but not active . Example date=((2010, 3, 20), (2012, 3, 20))) date=((2012, 3, 20), (2014, 3, 20))) date=((2014, 3, 20), (2016, 3, 20))) ... i should use bing and google to crawl