Running scrapy spider programmatically

To illustrate the use of running scrapy spiders from a program instead of using the scrapy command line. Read more about it in the blogpost here.

If you are really interested in running your scrapers programmatically please look at my new project arachne, it allows you to run your scraper spiders without the hassles of a program but from a python pip package