feat(agent): Add URL whitelisting/blacklisting to config and URL validation

[Satyam97]

Background

This PR refers to the solution for enhancement suggested here: https://github.com/Significant-Gravitas/AutoGPT/issues/5289

This basically takes account white/black listing of URLs during context setting in agents.

Changes 🏗️

I have added 2 env variables for web policy and URL list. With a combination of both, a URL is white or blacklisted by the read_webpage function.

PR Quality Scorecard ✨

• [x] Have you used the PR description template?   +2 pts
• [x] Is your pull request atomic, focusing on a single change?   +5 pts
• [x] Have you linked the GitHub issue(s) that this PR addresses?   +5 pts
• [x] Have you documented your changes clearly and comprehensively?   +5 pts
• [x] Have you changed or added a feature?   -4 pts
  • [x ] Have you added/updated corresponding documentation?   +4 pts
  • [ ] Have you added/updated corresponding integration tests?   +5 pts
• [x] Have you changed the behavior of AutoGPT?   -5 pts
  • [ ] Have you also run agbenchmark to verify that these changes do not regress performance?   +10 pts

[netlify[bot]]

✅ Deploy Preview for auto-gpt-docs ready!

Name	Link
🔨 Latest commit	27b1b4edabaaf44f8da4f0c306d9b0c209646b12
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/65cd180460115f0008405bfe
😎 Deploy Preview	https://deploy-preview-6848--auto-gpt-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[ntindle]

remove falcon from the PR

[github-actions[bot]]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[Swiftyos]

Closed due to a lack of activity since Feb. If you want to pick this up again let me know and we can reopen