autopager icon indicating copy to clipboard operation
autopager copied to clipboard

Published 20 hours ago • verified publisher icon TeamHG-Memex

detect pagination options links

Open kmike opened this issue 8 years ago • 0 comments

Websites often provide links "show 20/50/100 results per page"; by following them crawler can get the same contents multiple times. It'd be nice to detect these links.

kmike avatar Mar 17 '16 15:03 kmike