Sander Heilbron

icon indicating a website link https://www.sanderheilbron.nl

icon location The Netherlands icon location Senior Web Performance Consultant at Iron/Out. Creator of httpstatus.io.

Results 2 issues of Sander Heilbron

'Allow all' statement or blank line returns true for disallowed URLs

When a `Allow: /` statement or blank line precedes a `Disallow: /foo` statement, the `Disallow: /foo` statement always returns `true`. Unfortunately this does not match how Google interprets a robots.txt...

Save current URL in Wayback Machine

Example: https://web.archive.org/save/https://www.nu.nl/

enhancement