[Radar] Add crawlers section to glossary

[andre-j3sus]

Summary

Add crawlers section to Radar glossary

Screenshots (optional)

Documentation checklist

• [X] The documentation style guide has been adhered to.

[hyperlint-ai-deprecated[bot]]

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.

---

PR Change Summary

Added a new section on web crawlers to the Radar glossary, enhancing the documentation with relevant information about their function and usage.

• Introduced a new 'Web Crawlers' section in the glossary.
• Provided definitions and explanations of web crawlers and their purposes.
• Included details on the crawl-to-refer ratio metric.

Modified Files

• src/content/docs/radar/glossary.mdx

---

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

[github-actions[bot]]

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern	Owners
/src/content/docs/radar/	@tiagoad, @andre-j3sus, @cloudflare/pcx-technical-writing

[github-actions[bot]]

Preview URL: https://6b01e9ec.preview.developers.cloudflare.com Preview Branch URL: https://ajesus-radar-5595.preview.developers.cloudflare.com

Files with changes (up to 15)

Original Link	Updated Link
https://developers.cloudflare.com/radar/glossary/	https://ajesus-radar-5595.preview.developers.cloudflare.com/radar/glossary/

[andre-j3sus]

Merged with #22786