gotor icon indicating copy to clipboard operation
gotor copied to clipboard

Published 20 hours ago • verified publisher icon DedSecInside
→

Metadata

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Results 5 gotor issues
Sort by recently updated
recently updated
newest added

Run without tor?

1 comment

Should there be an option to run without using socks5 proxy?

KingAkeem avatar KingAkeem

question

Add cache to avoid excessive lookups

There are two methods of interacting with a node. - Crawl: traverse a node and it's children without storing them in memory - Load: stores and and it's children in...

KingAkeem avatar KingAkeem

enhancement
Not Available
performance
Hacktoberfest

Try using multiple tor circuits to achieve better performance

It should be possible to open multiple tor connections using different SOCKS/CONTROL ports. It may provide a performance boost to execute requests using different connections. How to open multiple connections:...

KingAkeem avatar KingAkeem

Low Priority

Handle 429 status code with retry

429 indicates too many requests and possibly has a header attached that indicates when another request should be retried `Retry-After`. This could be used to pull those requests into a...

KingAkeem avatar KingAkeem

enhancement
Medium Priority
performance
concurrency

Randomized Headers

4 comment

I found this wonderful snippet in `gocolly`, this file could make random headers rather simple: https://github.com/gocolly/colly/blob/master/extensions/random_user_agent.go

thehappydinoa avatar thehappydinoa

← Metadata

154
Stars
43
Forks
Watchers

Owner

verified publisher icon DedSecInside

Metadata

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Back