don't crawl files that haven't been modified since the last crawl

[rpendleton]

According to RFC 7232 §4.1, when a page is returned with a 304 Not Modified header, it is recommended that headers such as Content-Type should not be sent the the client again. When verbose logging in osg is enabled, this would manifest as not crawling a page because its content type was an empty string.

Instead of doing that, we should just log that the page hasn't been modified and avoid crawling it completely. This is okay since a 304 response isn't permitted to have a response body anyway, so there's no need to crawl it.