fetchbot
fetchbot copied to clipboard
PuerkitoBio
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Path matches still take priority, but ties are broken with the number of additional matching criteria. fix #32
Currently the Mux prioritizes the Handler with the most specific Path, but if multiple Handlers are registered with the same Path, but with different criteria such as ContentType, the selected...
Metadata
Owner
Metadata
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.