James Sheppard

I just pushed code to enable email alert capabilities (if desired). If you are on an older version of Malspider, you'll need to make a few changes. 1) Get the...

Add email alerts via the admin interface: Admin Panel -> Email Alerts -> "Add Email Alert" Provide a subject line for your email "ie. Malspider Email Summary", a list of...

Forgot one thing - you'll need to navigate to malspider_django/malspider_django/settings.py, at the bottom of the file is a variable for the EMAIL_HOST and EMAIL_PORT. Uncomment these and apply your settings.

Malspider generates "start" urls to crawl and they are all http. I can add https start urls fairly easily, but I don't know if that will fix the problem you...

Hi Marcus, Thank you for following up on this. I fixed a bug that was causing the spider to not follow 301/302 redirects, but I haven't committed the code yet....

Thank you for your patience. I pushed the code. Redirects are enabled and https is priortized in the list of start_urls, but again, this doesn't mean an http page will...

Hey guys, Thank you for the feedback. Malspider was originally tested on minimal installs of CentOS 6, CentOS 7, and Ubuntu 14. The installation script worked flawlessly, but based on...

Thank you for the suggestion. I'll add this to my list for this week. On Sat, Feb 25, 2017 at 8:39 AM, vletoux wrote: > The only_full_group_by mode is a...

Are there blacklists/feeds you recommend? This is something I'd like to put in the dev pipeline for a later release. On Tue, Jun 14, 2016 at 11:19 PM, r3comp1le [email protected]...

Hey Alex, New jobs are still launched, but scrapyd (according to the default settings) will only keep 10k finished jobs in the launcher. This means once you reach 10k, the...