Alexander Sibiryakov
Alexander Sibiryakov
In general looks awesome! It definitely requires some small changes, but this is already a great contribution!
So tests are broken, we either have to find a way how to test it with Travis CI, or disable this test for now.
> The issue is that until the Spider finishes its current batch, the DBWorker will just keep sending new ones. In my case, the DBWorker have time to flush the...
I'm marking these PR as WIP. Meaning it's Work in Progress, and we shouldn't merge it. OK?
Hey @wpxgit what do you think of all that? Do you plan to contribute more?
I haven't heard anything @maisumbruno . At Scrapinghub we're fine with HBase so far.
@maisumbruno Definitely. I would recommend to inspire from HBaseBackend, where you can find a queue suitable for large scale crawling. Also you can start implementing it by parts, say first...
Hi @grammy-jiang it's quite an interesting finding. The thing is Frontera tries to be both a distributed and non-distributed crawl frontier framework. And backend became a place in internal architecture...
Please remove reference to bigbot_common
Yeah, I absolutely agree with adding this field.