boudicca.events icon indicating copy to clipboard operation
boudicca.events copied to clipboard

Published 20 hours ago • verified publisher icon boudicca-events

automatic robots.txt parsing in fetcher

Open kadhonn opened this issue 1 year ago • 1 comments

in order to be nice and complying we should do this

kadhonn avatar Feb 19 '24 07:02 kadhonn

don't forget about the meta-robots tag!

kadhonn avatar Mar 08 '24 19:03 kadhonn