colly icon indicating copy to clipboard operation
colly copied to clipboard

Is it possible to swap the scrape code, to not use http package?

Open Jleagle opened this issue 2 years ago • 2 comments

Recently I have been having to scrape sites using a headless chrome to parse the DOM after JS changes. Also it helps get through Cloudflare etc, is there anyway i can swap out the scrape function to use my own scraper?

Thanks!

Jleagle avatar Oct 30 '21 16:10 Jleagle

You can certainly implement http.RoundTripper and plug it into Colly with WithTransport.

WGH- avatar Nov 07 '21 17:11 WGH-

Great I didn't think about that, cheers!

Jleagle avatar Nov 07 '21 22:11 Jleagle