colly
colly copied to clipboard
Is it possible to swap the scrape code, to not use http package?
Recently I have been having to scrape sites using a headless chrome to parse the DOM after JS changes. Also it helps get through Cloudflare etc, is there anyway i can swap out the scrape function to use my own scraper?
Thanks!
You can certainly implement http.RoundTripper
and plug it into Colly with WithTransport
.
Great I didn't think about that, cheers!