Handle Config errors with great messages
The Config file is the most error-prone part of the process from the user's perspective. However, we can't really get around this — there are just a lot of choices to make. Let's make it as easy as possible to interpret errors.
I think this is a good case for test-driven development. I'll start by creating some pathologically flawed config files — though really each can only trigger one error, so I may have to think through which errors to embed.
Just experienced a good case that could use special handling: a config file with no From field, when in spider mode. Spider mode is useless without From, so crawl should check for it.
Are there other mandatory fields (possibly different in each mode) that should be checked for?
Does Go's unmarshalling pattern allow for this to be easily checked?