Add custom headers support and also nonrender

[lee-hodg]

Implements

• Allow custom headers setting, which is useful for some sites you may wish to scrape
• Also when doing a crawl we may want the process url to have the &norender parameter to stop javascript rendering to speed up performance of the crawl, so here allow for that.