Split Google crawlers into categories

[alaz]

https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot

[aydinkazim]

Hey, I think one possible fix for this could be adding Chrome-Lighthouse to the list of Google crawlers. Since it's commonly used for performance audits, it might make sense to group it with the other Google bots. Here's what the updated lib/legitbot/google.rb could look like:

rule Legitbot::Google, %w[
  APIs-Google
  AdsBot-Google-Mobile
  AdsBot-Google
  Googlebot
  Mediapartners-Google
  AdsBot-Google-Mobile-Apps
  FeedFetcher-Google
  Google-Read-Aloud
  DuplexWeb-Google
  googleweblight
  Storebot-Google
  Google-Site-Verification
  Google-InspectionTool
  Chrome-Lighthouse
]

[alaz]

Hi, the idea of the card is a bit different, I was thinking of packaging Google crawler User-Agents into few lists just like Google presents them on this page.

I agree with your suggestion. Please feel free to create a PR.

[aydinkazim]

Ah, now I understand 😮‍💨 I think you want to categorize it into three groups as mentioned in the document. Still, thank you for your kindness; I have created the PR. #160