zimit
zimit copied to clipboard
Automatically add exclusion rules based on `robots.txt`
See https://github.com/webrecorder/browsertrix-crawler/issues/631