Nirbhay Kundan

Custom feed will look like this python kafka_monitor.py feed { "url": "http://dmoztools.net", "appid": "testapp", "crawlid": "ABC123", "spiderid": "myspiderid", "headers": { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,ima/webp,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "X-Requested-With": "dmoztools.net", "User-Agent": "My Custom...

Due to shared cookie middle ware the coverage has decreased by 0.4%. @madisonb do you think this can be managed? I improved few percentage in distributed_scheduler.

Working towards it. I made the custom request working with headers and cookies. Working on shared cookie instances, shared via redis seperated by crawl/spider ids

Below custom cookie middleware worked for me. Not sure if this is a right place to initiate redis_conn. Could not find a way to share `DistributedScheduler` redis_conn. ` import redis...