Crawlpy

Python web spider/crawler based on scrapy with support for POST/GET login, variable level of recursions/depth and optionally save to disk.

Requirements

• python 2.7
• lxml
• pip

pip install Scrapy

Features

• POST/GET Login prior crawling
• Can handle logins that requires dynamic CSRF token
• Variable level of crawling depth
• Optionally save webpages to disk

Roadmap

Find all planned features and their stati here: https://github.com/cytopia/crawlpy/issues/1

Usage

# stdout output
scrapy crawl crawlpy -a config=/path/to/crawlpy.config.json

# save as json (url:, status:, depth:, referer:) to 'urls.json'
scrapy crawl crawlpy --loglevel=INFO -a config=/path/to/crawlpy.config.json -o urls.json -t json

# save as csv (url, status, depth, referer) to 'urls.csv'
scrapy crawl crawlpy --loglevel=INFO -a config=/path/to/crawlpy.config.json -o urls.csv -t csv

Configuration

Make a copy of crawlpy.config.json-sample (e.g.: example.com-config.json) and adjust the values accordingly.

Note: It must be a valid json file (without comments), otherwise crawlpy will throw errors parsing json. (Use http://jsonlint.com/ to validate your config file.)

{
    "proto": "http",        // 'http' or 'https'
    "domain": "localhost",  // Only the domain. e.g.: 'example.com' or 'www.example.com'
    "depth": 3,             // Nesting depth to crawl
    "ignores": [],          // Array of substrings to deny/ignore when found in URL
	"httpstatus_list": [],  // Array of http status codes to handle (default is 2xx)
    "login": {              // Login section
        "enabled": false,   // Do we actually need to do a login?
        "method": "post",   // 'post' or 'get'
        "action": "/login.php", // Where the post or get will be submitted to
        "failure": "Password is incorrect", // The string you will see on login failure
        "fields": {         // POST/GET Fields to submit to login page
            "username": "john",
            "password": "doe"
        },
        "csrf": {
            "enabled": false, // Login requires a CSRF token?
            "field": "csrf"   // Input field name that holds dynamic CSRF token
        }
    },
    "store": {              // Store section
        "enabled": false,   // save to disk?
        "path": "./data"    // path for saving (rel or abs)
    }
}

Detailed description

Note about CSRF

Scrapy will most likely handle this automatically, so its best to turn off custom csrf in the config. If there is however any situation where the built-in CSRF recognition does not work, try the user-defined one. If none of them work, drop me an issue.

Reference

• https://stackoverflow.com/questions/5851213/crawling-with-an-authenticated-session-in-scrapy
• https://stackoverflow.com/questions/38619807/scrapy-recursive-link-crawler-with-login-help-me-improve/
• http://thuongnh.com/building-a-web-crawler-with-scrapy/

License

MIT License

Copyright (c) 2016 cytopia