web_selenium: add ignore local proxy argument for ChromeDriver

[LawrenceHan]

Background

This PR is intend to fix some issue with error message browse_website returned: Error: Message. When using a local proxy, the ChromeDriver returns 502 error code (but not visiable in the log). In fact driver opens a separate chrome app (with a default profile) that uses the system proxy, so here we add the ignore argument that is only protect the driver initialization process (which is elegant).

Changes

options.ignore_local_proxy_environment_variables(), in web_selenium.py, under the scrape_text_with_selenium function.

Documentation

Only comments added.

Test Plan

0. MacOS
1. Add your local proxy in terminal export https_proxy=http://127.0.0.1:7890 http_proxy=http://127.0.0.1:7890 all_proxy=socks5://127.0.0.1:7890.
2. In terminal, run python -m autogpt.
3. When browse_website executed, error SYSTEM: Command browse_website returned: Error: Message: logged.

More details:

Caught an exception on chromium/webdriver.py line: 100:

        try:
            RemoteWebDriver.__init__(
                self,
                command_executor=ChromiumRemoteConnection(
                    remote_server_addr=self.service.service_url,
                    browser_name=browser_name, vendor_prefix=vendor_prefix,
                    keep_alive=keep_alive, ignore_proxy=_ignore_proxy),
                options=options)
        except Exception:
            self.quit()
            raise
        self._is_remote = False

Actuall file caused the exception: remote_connection.py line: 400:

                try:
                    data = utils.load_json(data.strip())
                except ValueError:
                    if 199 < statuscode < 300:
                        status = ErrorCode.SUCCESS
                    else:
                        status = ErrorCode.UNKNOWN_ERROR
                    return {'status': status, 'value': data.strip()}

Method: POST URL: http://localhost:62385/session Body: {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "args": ["user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36", "--no-sandbox", "--headless", "--disable-gpu"]}}}} Headers: {'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Type': 'application/json;charset=UTF-8', 'User-Agent': 'selenium/4.1.4 (python mac)'}

Response: "Remote response: status=502 | data= | headers=HTTPHeaderDict({'Connection': 'close', 'Content-Length': '0'})"

PR Quality Checklist

• [x] My pull request is atomic and focuses on a single change.
• [x] I have thoroughly tested my changes with multiple different prompts.
• [x] I have considered potential risks and mitigations for my changes.
• [x] I have documented my changes clearly and comprehensively.
• [x] I have not snuck in any "extra" small tweaks changes

[hdkiller]

I was trying to reproduce your issue which this supposed to fix on Linux without applying this PR.

I've did an export http_proxy="http://192.." to set my proxy then launched autgpt which was able to use the browse_website with chrome.

[LawrenceHan]

I was trying to reproduce your issue which this supposed to fix on Linux without applying this PR.

I've did an export http_proxy="http://192.." to set my proxy then launched autgpt which was able to use the browse_website with chrome.

Firstly, thank you for replying. It was my bad to not given the full context here. The docker (or Linux) is fine with it, the issue was happened on my M1 Mac. Let me rewrite the test plan, just a min.

[LawrenceHan]

@hdkiller Updated with full context, let me know if there's any misunderstandings.

[hdkiller]

Thank you!

Unfortunately I am not able to test this on an M1 mac right now. I've added the relevant labels, hopefully someone can confirm the issue and your resolution.

[LawrenceHan]

@hdkiller Y

Thank you!

Unfortunately I am not able to test this on an M1 mac right now. I've added the relevant labels, hopefully someone can confirm the issue and your resolution.

Ah, about that M1 thing, turns out it doesn't related to M1. You can simply test it on MacOS.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 27, 2023 8:17am

[p-i-]

This is a mass message from the AutoGPT core team. Our apologies for the ongoing delay in processing PRs. This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to: https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting