crawler-python

crawler-python copied to clipboard

the yelp crawler is too slow

1

need optimize the review crawling parts (i.e., reviews_in_page function).

hezila

Oops, after testing, yelp blocks tor and goengine. So alternative proxy solution is needed (dynamic changing ip address is preferred).

hezila

support distribution crawling

hezila

Errors: No handlers could be found for logger....

hezila