robots.js icon indicating copy to clipboard operation
robots.js copied to clipboard

'Allow all' statement or blank line returns true for disallowed URLs

Open sanderheilbron opened this issue 9 years ago • 0 comments

When a Allow: / statement or blank line precedes a Disallow: /foo statement, the Disallow: /foo statement always returns true.

Unfortunately this does not match how Google interprets a robots.txt file.

sanderheilbron avatar Mar 21 '15 21:03 sanderheilbron