robots.js
robots.js copied to clipboard
'Allow all' statement or blank line returns true for disallowed URLs
When a Allow: /
statement or blank line precedes a Disallow: /foo
statement, the Disallow: /foo
statement always returns true
.
Unfortunately this does not match how Google interprets a robots.txt file.