Alexander Sibiryakov
Alexander Sibiryakov
We need to adapt interfaces in `FronteraManagerWrapper`, `FronteraManager` and `Backend`. I think we need to propagate type (error happened during response processing), response itself, along with error structure.
Can you fork Frontera to your own account, and use your local branch?
It would be good to improve performance on that `safe_url_string` and use it on `Request` instantiation, probably.
@lljrsr This is mostly done because there is no clear way of controlling consumption rate on producer side in ZeroMQ. Moreover the contents of queue on ZeroMQ side and the...
Hi Canh, It's not a bug, HBaseBackend implies distributed backend and spiders run mode, therefore it requires strategy worker and scoring log set up. Отправлено с iPhone > 19 июня...
@nautilus28 this is going to be added with upcoming kafka-python 1.x.x support release.
@plafl @kalessin please have a look.
Great, I think I'll do some docs and tests.
@bbotella Honestly, I don't think people will be using MongoDB backend because of performance, instead because of ease of development and schemaless concept. Therefore, I would stay with most transparent...
Hi, @ZipFile > Spiders distributed by availability zones. In order to utilize full throughput during broad crawls it makes sense to spread some of your spiders across different physical locations....