Be able to browse_website protected by cloudflare

[hugo4711]

Duplicates

• [X] I have searched the existing issues

Summary 💡

A website that is protected by cloudflare cannote be browsed without an error. We need a longer timeout.

Examples 🌈

NEXT ACTION: COMMAND = browse_website ARGUMENTS = {'url': 'https://SITE', 'question': 'Please evaluate the design, content, and user experience of businesswarrior.de. Specifically, please provide feedback on the layout, color scheme, typography, images, and content. Please also identify any usability issues and suggest improvements.'} Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for ... Input:y -=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-= Text length: 555 characters Adding chunk 1 / 1 to memory Summarizing chunk 1 / 1 Added chunk 1 summary to memory Summarized 1 chunks. SYSTEM: Command browse_website returned: ("Answer gathered from website: The given text does not provide enough information to evaluate the design, content, and user experience of businesswarrior.de. It only mentions that the website is being checked for security and may require enabling JavaScript and cookies to continue. \n \n Links: ['Cloudflare (https://www.cloudflare.com?utm_source=challenge&utm_campaign=j)']", <selenium.webdriver.chrome.webdriver.WebDriver (session="3e70f06d4fc3f98fa515ab73e066c2bb")>)

Motivation 🔦

Websites which have a Cloudflare protection (DDoS, without the need to click something) cannot be browsed otherwise, limiting the abilities of AutoGPT

[k-boikov]

Might improve with https://github.com/Significant-Gravitas/Auto-GPT/pull/2299 Might be solved by https://github.com/Significant-Gravitas/Auto-GPT/pull/2170

[ntindle]

Might improve with https://github.com/Significant-Gravitas/Auto-GPT/pull/2299 Might be solved by https://github.com/Significant-Gravitas/Auto-GPT/pull/2170