SimonC-Audigent

Add explicit disallow feature

2

There are some scenarios, where we want to check if the robots.txt is explicitly disallowing the UA (wildcard not included). This a common behaviour for ads crawlers such as Google...