goclone icon indicating copy to clipboard operation
goclone copied to clipboard

Published 20 hours ago • verified publisher icon imthaghost

Recursive clone

Open W-Lawless opened this issue 1 year ago • 2 comments

I realize this might be dangerous if someone haphazardly attempts WIkipedia or something, but it would be nice to have an -r flag to recursively follow all tags and clone subpages

W-Lawless avatar Jun 16 '23 16:06 W-Lawless

This sounds more like scrapping, no?

mesaglio avatar Jun 17 '23 16:06 mesaglio

This sounds more like scrapping, no?

I apologize, yesterday when I was looking through here and searching github for a tool, I got very limited results for some reason, although now attempting the same search again there are over 17k repos for this. I swapped the scrape keyword for clone trying to find better results.

That being said, is there any repo you recommend for this in particular? I'd like to essentially clone/scrape a subreddit-style website and make a lot of the info there searchable / indexable. A lot of discussion on programming/opsec stuff that could be made much more actionable if I can search it in ways the exceed the standard sidebar search input.

Thanks!

W-Lawless avatar Jun 17 '23 16:06 W-Lawless