AutoGPT
AutoGPT copied to clipboard
Significant-Gravitas
Error if Command: browse_website is run within a dev container
Duplicates
- [X] I have searched the existing issues
Steps to reproduce 🕹
- Clone github repository of autoGPT, latest or stable. Doesn't matter.
- Install dev container : https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers 3, Have docker desktop up and running.
- In vscode use CTRL+SHIFT+P and search for Dev Containers: Reopen in Container.
- After the dev container is opened in VSCode and running on docker, do pip install -r requirements.txt
- Start the autgpt using the command: python -m autogpt
Current behavior 😯
Once when it tries to browse a website with the command : browse_website and Arguments : {'url':
it returns an error in the System message:
SYSTEM: Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace: #0 0x55ac2df50fe3 <unknown> #1 0x55ac2dc8fd36 <unknown> #2 0x55ac2dcb6f4a <unknown> #3 0x55ac2dcb4a9b <unknown> #4 0x55ac2dcf6af7 <unknown> #5 0x55ac2dcf611f <unknown> #6 0x55ac2dced693 <unknown> #7 0x55ac2dcc003a <unknown> #8 0x55ac2dcc117e <unknown> #9 0x55ac2df12dbd <unknown> #10 0x55ac2df16c6c <unknown> #11 0x55ac2df204b0 <unknown> #12 0x55ac2df17d63 <unknown> #13 0x55ac2deeac35 <unknown> #14 0x55ac2df3b138 <unknown> #15 0x55ac2df3b2c7 <unknown> #16 0x55ac2df49093 <unknown> #17 0x7f0432fd2ea7 start_thread
Expected behavior 🤔
It should be able to run the browse_website command normally and browse the website with the question.
Your prompt 📝
# Paste your prompt here
The error you are encountering is related to Chrome not starting correctly when using Selenium WebDriver. You can try adding a few more arguments to the options object to resolve this issue.
Update the scrape_text_with_selenium(url) function in your script as follows:
File: autogpt/web.py:
def scrape_text_with_selenium(url):
logging.getLogger("selenium").setLevel(logging.CRITICAL)
options = Options()
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)
# Add the following lines:
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(
executable_path=ChromeDriverManager().install(), options=options
)
The error you are encountering is related to Chrome not starting correctly when using Selenium WebDriver. You can try adding a few more arguments to the options object to resolve this issue.
Update the scrape_text_with_selenium(url) function in your script as follows:
File: autogpt/web.py:
def scrape_text_with_selenium(url): logging.getLogger("selenium").setLevel(logging.CRITICAL) options = Options() options.add_argument( "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36" ) # Add the following lines: options.add_argument("--no-sandbox") options.add_argument("--headless") options.add_argument("--disable-dev-shm-usage") driver = webdriver.Chrome( executable_path=ChromeDriverManager().install(), options=options )
I added it, but it didn't seem to fix the problem unfortunately. Maybe it has something to do with the pathing? I don't know what the file system usually looks like in a docker container? ` options.binary_location = "/usr/local/bin/chromedriver" also probably wrong file name but it's an example
I've got this same issue. Tried the suggested arguments to add to options as well as options.binary_location = "C:\\Users\\[user]\\.wdm\\drivers\\chromedriver\\win32\\112.0.5615.49\\chromedriver.exe" to no avail. Same error persists
Make sure chrome is actually installed in your container, the current Dockerfile doesn't install it for example.
Running in a Win10 vm, not a docker container.
I'm getting some traction with
`
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(
# executable_path=ChromeDriverManager().install(), options=options
executable_path="C:\\Users\\[user]\\.wdm\\drivers\\chromedriver\\win32\\112.0.5615.49\\chromedriver.exe", options=options
)
`
Still getting some issues:
`
DevTools listening on ws://127.0.0.1:52210/devtools/browser/92ead2e2-e12e-4d14-93fd-1ea445cf6fd5 [0415/173857.330:ERROR:command_buffer_proxy_impl.cc(128)] ContextResult::kTransientFailure: Failed to send GpuControl.CreateCommandBuffer. [0415/173858.212:INFO:CONSOLE(1)] "Unrecognized Content-Security-Policy directive 'prefetch-src'.", source: https://www.viator.com/Italy-tours/Attraction-Tickets/d57-g8-c29 (1) [0415/173858.527:INFO:CONSOLE(2)] "Unrecognized Content-Security-Policy directive 'prefetch-src'.", source: https://js.datadome.co/tags.js (2) [0415/173858.543:INFO:CONSOLE(2)] "Error", source: https://js.datadome.co/tags.js (2) [0415/173858.565:INFO:CONSOLE(2)] "Error", source: https://js.datadome.co/tags.js (2) [0415/173858.797:INFO:CONSOLE(1)] "Unrecognized Content-Security-Policy directive 'prefetch-src'.", source: https://cache.vtrcdn.com//orion/js/vendors~00cb062a.33a29d6c69590b570e1d.chunk.js (1) [0415/173858.938:INFO:CONSOLE(1)] "Evidon -- evidon-notice-link not found on page, cant display the consent link.", source: https://c.evidon.com/sitenotice/evidon-sitenotice-tag.js (1) [0415/173859.163:INFO:CONSOLE(41)] "Unrecognized Content-Security-Policy directive 'prefetch-src'.", source: https://www.googletagmanager.com/gtag/js?id=DC-12197993&l=dataLayer&cx=c (41) [0415/173859.163:INFO:CONSOLE(41)] "Unrecognized Content-Security-Policy directive 'prefetch-src'.", source: https://www.googletagmanager.com/gtag/js?id=DC-12197993&l=dataLayer&cx=c (41) `
@aeiberra comments worked along with :
sudo apt-get update
sudo apt-get install wget gnupg
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
sudo apt-get update
sudo apt-get install google-chrome-stable
looks like web.py got removed from Stable in a recent merge. This broken again and the error persists even with the commands from @darrynv
SYSTEM: Command browse_website returned: Error: Message: unknown error: Chrome failed to start: crashed. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x560011b04fe3 <unknown> #1 0x560011843d36 <unknown> #2 0x56001186cb20 <unknown> #3 0x560011868a9b <unknown> #4 0x5600118aaaf7 <unknown> #5 0x5600118aa11f <unknown> #6 0x5600118a1693 <unknown> #7 0x56001187403a <unknown> #8 0x56001187517e <unknown> #9 0x560011ac6dbd <unknown> #10 0x560011acac6c <unknown> #11 0x560011ad44b0 <unknown> #12 0x560011acbd63 <unknown> #13 0x560011a9ec35 <unknown> #14 0x560011aef138 <unknown> #15 0x560011aef2c7 <unknown> #16 0x560011afd093 <unknown> #17 0x7f4d24f1dea7 start_thread
I have this error in latest pull
NEXT ACTION: COMMAND = browse_website ARGUMENTS = {'url': 'website_url', 'question': 'what information to extract from website'}
SYSTEM: Command browse_website returned: Error: Message: invalid argument (Session info: chrome=112.0.5615.121) Stacktrace: #0 0x559ce5494fe3 <unknown> #1 0x559ce51d3bc1 <unknown> #2 0x559ce51be446 <unknown> #3 0x559ce51bc7f3 <unknown> #4 0x559ce51bcc3d <unknown> #5 0x559ce51d5b16 <unknown> #6 0x559ce524a8c5 <unknown> #7 0x559ce52318c2 <unknown> #8 0x559ce524a232 <unknown> #9 0x559ce5231693 <unknown> #10 0x559ce520403a <unknown> #11 0x559ce520517e <unknown> #12 0x559ce5456dbd <unknown> #13 0x559ce545ac6c <unknown> #14 0x559ce54644b0 <unknown> #15 0x559ce545bd63 <unknown> #16 0x559ce542ec35 <unknown> #17 0x559ce547f138 <unknown> #18 0x559ce547f2c7 <unknown> #19 0x559ce548d093 <unknown> #20 0x7f9f13a94b43 <unknown>
Modifying web_selenium.py with the options per @darkcount2011 and installing chrome per @darrynv has got me past the errors. Though the options are for chrome, if anyone is getting these errors with Safari or Firefox the options would need to be modified somewhere else.
For me, the Chat-GPT fixed the problem in web_selenium.py file:
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--headless")
I am using headless server for this.
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:
def scrape_text_with_selenium(url):
logging.getLogger("selenium").setLevel(logging.CRITICAL)
options = Options()
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)
# Add the following lines:
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(
executable_path=ChromeDriverManager().install(), options=options
)
Plus running the commands:
sudo apt-get update sudo apt-get install wget gnupg wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add - sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' sudo apt-get update sudo apt-get install google-chrome-stable
Posted by @darrynv
Solved for me, running on Kali Linux Subsystem for windows.
PS: last commands produced the following errors:
Failed to retrieve available kernel versions.
Failed to check for processor microcode upgrades.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
But browse_website COMMAND works now. Thanks for the continuous support!
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011: ...
this all worked for me. I am also on macOS
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:
Change this part of code:
else:
if platform == "linux" or platform == "linux2":
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--no-sandbox") # indent this line
options.add_argument("--headless") # Add this line
driver = webdriver.Chrome(
executable_path=ChromeDriverManager().install(), options=options
)
driver.get(url)
Work for me on docker container
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011:
Change this part of code:
else: if platform == "linux" or platform == "linux2": options.add_argument("--disable-dev-shm-usage") options.add_argument("--remote-debugging-port=9222") options.add_argument("--no-sandbox") # indent this line options.add_argument("--headless") # Add this line driver = webdriver.Chrome( executable_path=ChromeDriverManager().install(), options=options ) driver.get(url)Work for me on docker container
I'll try again and see if it works, thanks
Grafting together various responses and threads, my web_selenium.py now looks like this and it finally runs without any errors. Windows 10, running as is
` options = options_availableCFG.selenium_web_browser options.add_argument( "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36" )
if CFG.selenium_web_browser == "firefox":
driver = webdriver.Firefox(
executable_path=GeckoDriverManager().install(), options=options
)
elif CFG.selenium_web_browser == "safari":
# Requires a bit more setup on the users end
# See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
driver = webdriver.Safari(options=options)
else:
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--ignore-ssl-errors=true")
options.add_argument("--ignore-certificate-errors")
options.binary_location = "F:\Program Files\Google\Chrome\Application\chrome.exe"
driver = webdriver.Chrome(
#executable_path=ChromeDriverManager().install(), options=options
executable_path="F:\user\Downloads\chromedriver_win32\chromedriver.exe", options=options
)
driver.get(url)
`
I think this is a specific problem for M1 or other ARM processor users. I assume you also have a problem with macOS. Try setting up Docker to use Rosetta for x86/amd64 emulation on Apple Silicon
For me, the Chat-GPT fixed the problem in web_selenium.py file:
options.add_argument("--no-sandbox") options.add_argument("--disable-dev-shm-usage") options.add_argument("--remote-debugging-port=9222") options.add_argument("--headless")I am using headless server for this.
I add upper code to autogpt/commands/web_selenium.py, but still have problem.
SYSTEM: Command browse_website returned: Error: Message: unknown error: Chrome failed to start: exited abnormally. (chrome not reachable) (The process started from chrome location /usr/bin/chromedriver is no longer running, so ChromeDriver is assuming that Chrome has crashed.) Stacktrace: #0 0x5555f49b8ba3 <unknown> #1 0x5555f461632c <unknown> #2 0x5555f463c58e <unknown> #3 0x5555f4638998 <unknown> #4 0x5555f4678ece <unknown> #5 0x5555f4670223 <unknown> #6 0x5555f4642d13 <unknown> #7 0x5555f46441e2 <unknown> #8 0x5555f498a663 <unknown> #9 0x5555f498cfa2 <unknown> #10 0x5555f498c980 <unknown> #11 0x5555f498d705 <unknown> #12 0x5555f4993316 <unknown> #13 0x5555f498da56 <unknown> #14 0x5555f496b0a4 <unknown> #15 0x5555f49a6a18 <unknown> #16 0x5555f49a6bae <unknown> #17 0x5555f49b337f <unknown> #18 0x7f1dc012b802 start_thread
below is the function scrape_text_with_selenium code:
def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
"""Scrape text from a website using selenium
Args:
url (str): The url of the website to scrape
Returns:
Tuple[WebDriver, str]: The webdriver and the text scraped from the website
"""
logging.getLogger("selenium").setLevel(logging.CRITICAL)
options_available = {
"chrome": ChromeOptions,
"safari": SafariOptions,
"firefox": FirefoxOptions,
}
options = options_available[CFG.selenium_web_browser]()
options.binary_location = "/usr/bin/chromedriver"
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--headless")
if CFG.selenium_web_browser == "firefox":
driver = webdriver.Firefox(
executable_path=GeckoDriverManager().install(), options=options
)
elif CFG.selenium_web_browser == "safari":
# Requires a bit more setup on the users end
# See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
driver = webdriver.Safari(options=options)
else:
if platform == "linux" or platform == "linux2":
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument('--headless')
options.add_argument("--no-sandbox")
if CFG.selenium_headless:
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(
executable_path="/usr/bin/chromedriver", options=options
)
#driver = webdriver.Chrome(
# executable_path=ChromeDriverManager().install(), options=options
#)
driver.get(url)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
# Get the HTML content directly from the browser's DOM
page_source = driver.execute_script("return document.body.outerHTML;")
soup = BeautifulSoup(page_source, "html.parser")
for script in soup(["script", "style"]):
script.extract()
text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = "\n".join(chunk for chunk in chunks if chunk)
return driver, text
@busyfree please try with the latest version, and if it's still broken, provide more info about how you are running Auto-GPT, e.g. what OS/environment.
This is the exact error I'm getting:
-=-=-=-=-=-=-= COMMAND AUTHORISED BY USER -=-=-=-=-=-=-=
[WDM] - Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 11.0MB/s]
SYSTEM: Command browse_website returned: Error: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255
| Thinking...
Which seems to be a little different from yours.. more:
╰─ dco exec auto-gpt python -it
Python 3.10.11 (main, Apr 12 2023, 11:49:01) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from selenium import webdriver
>>> from webdriver_manager.chrome import ChromeDriverManager
>>>
>>> driver = webdriver.Chrome(ChromeDriverManager().install())
[WDM] - Downloading: 100%|███████| 6.75M/6.75M [00:00<00:00, 24.8MB/s]
<stdin>:1: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriver.py", line 80, in __init__
super().__init__(
File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/chromium/webdriver.py", line 101, in __init__
self.service.start()
File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 106, in start
self.assert_process_still_running()
File "/home/appuser/.local/lib/python3.10/site-packages/selenium/webdriver/common/service.py", line 119, in assert_process_still_running
raise WebDriverException(f"Service {self.path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /home/appuser/.wdm/drivers/chromedriver/linux64/112.0.5615.49/chromedriver unexpectedly exited. Status code was: 255
@busyfree please try with the latest version, and if it's still broken, provide more info about how you are running Auto-GPT, e.g. what OS/environment.
I have update to lastest commit(2f053fe9db74a59001715a64791f1c983fbd46a0), but still have problem.
NEXT ACTION: COMMAND = browse_website ARGUMENTS = {'url': 'https://github.com/autogpt/autogpt', 'question': 'What is Auto GPT?'}
[WDM] - Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.75M/6.75M [00:00<00:00, 11.0MB/s]
SYSTEM: Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace: #0 0x55648d2a0fe3 <unknown> #1 0x55648cfdfd36 <unknown> #2 0x55648d006f4a <unknown> #3 0x55648d004a9b <unknown> #4 0x55648d046af7 <unknown> #5 0x55648d04611f <unknown> #6 0x55648d03d693 <unknown> #7 0x55648d01003a <unknown> #8 0x55648d01117e <unknown> #9 0x55648d262dbd <unknown> #10 0x55648d266c6c <unknown> #11 0x55648d2704b0 <unknown> #12 0x55648d267d63 <unknown> #13 0x55648d23ac35 <unknown> #14 0x55648d28b138 <unknown> #15 0x55648d28b2c7 <unknown> #16 0x55648d299093 <unknown> #17 0x7fd095e33802 start_thread
OS:AlmaLinux release 9.1 (Lime Lynx)
Thanks for posting; please move further discussion about M1-specific issues to #2600
I tried to add the modifications to posted by @darkcount2011 @dods30 in a Mac 2018 (not M1) using docker run and Im still receiving the error.
I tried to execute Auto-GPT with the goal to fix it is own browse_website function, but didn't work 😂
@Robsonsjre no use applying those mods as a fix is already in master and stable. If you're still having issues, please post a full log and device/version/platform info in the issue mentioned above.
https://github.com/Significant-Gravitas/Auto-GPT/issues/2600#issuecomment-1518220293 This solution worked for me
@aeiberra comments worked along with :
sudo apt-get update sudo apt-get install wget gnupg wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add - sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' sudo apt-get update sudo apt-get install google-chrome-stable
Thank you this did the trick for me as well
