scrapy-cluster

scrapy-cluster copied to clipboard

shivaabed1994

rafaharo

abhalawat

use ui mode exception stack info,but browser normal File "ui_service.py", line 121, in _kafka_stats r = requests.post(self.settings['REST_HOST'] + "/feed", json=data) File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 110, in post return request('post', url, data=data,...

new-wxw

Upgrading the ELK stack

2

Great project, thanks for sharing - and supporting for so long! I ran into a few problems running the ELK stack - the Elasticsearch container kept restarting with `java.lang.IllegalStateException` [docker-elk-logs.txt](https://github.com/istresearch/scrapy-cluster/files/6015465/docker-elk-logs.txt)...

4OH4

TypeError: can't pickle thread.lock objects

2

Hi. I don't know how much it happens or how much it already happenned but one of my crawl fell on the error below. I ran a thousand requests and...

benjaminelkrieff

Future of the project

9

Hi. I've just come across this project and it is exactly what we need. However, I've noticed there haven't been any updates for a while now. Could you guys please...

demisx

Python 3

6

Issue to register the request to migrate fully to Python 3.

deed02392

Adding support of shared cookies for requests having the same crawlid

3

Adding a custom download middleware distributed_cookies.DistributedCookiesMiddleware that uses Redis to set and get the serialized cookie. Adding a custom spider middleware distributed_cookies.ClearCookiesMiddleware that delete cookie if an item with the...

MohamedMb

UI for displaying information about Cluster

22

We need a small stand-alone web UI that ties in with the rest components in #24 to visualize the data generated by the cluster. You should also be able to...

madisonb