crawler-user-agents icon indicating copy to clipboard operation
crawler-user-agents copied to clipboard
verified publisher icon monperrus

adding IP ranges along with user-agent

Open maxadamo opened this issue 9 months ago • 1 comments

I don't know how feasible is this request, and I don't how easily it can be implemented, but I thought that the user agent header is there to be forged.

For instance, in the case of Google, there is this page: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot

and this Json contains the IPs: https://developers.google.com/search/apis/ipranges/googlebot.json

maxadamo avatar Mar 10 '25 15:03 maxadamo

it's a good idea. pull-request welcome!

monperrus avatar Mar 11 '25 07:03 monperrus

FTR ip range of TikTok spider

From ttspider-feedback [email protected] The IP range is 47.128.0.0/16

monperrus avatar Apr 25 '25 12:04 monperrus