Robots.txt-Parser-Class
Robots.txt-Parser-Class copied to clipboard
[BUG] Allowed in, while the only rule specifies Disallow: /
$t1gor = new t1gor\RobotsTxtParser\RobotsTxtParser(<<<TXT
user-agent:*
disallow:/
TXT
);
var_dump($t1gor->isAllowed('/'));
Output: bool(true)
Expected: bool(false)
Quote
Valid robots.txt lines consists of a field, a colon, and a value. Spaces are optional, but recommended to improve readability.
Source: https://developers.google.com/search/docs/advanced/robots/robots_txt#syntax