hop icon indicating copy to clipboard operation
hop copied to clipboard
verified publisher icon seanmor5

Draft: Add robots support and test

Open acalejos opened this issue 1 year ago • 0 comments

Adds support for robots.txt and a small test harness to test crawling.

Right now the test harness just tests the default validations and the crawlers ability to move between links.

acalejos avatar Jun 30 '24 04:06 acalejos