Implement Web Browser using Playwright

[yimothysu]

Currently, OpenDevin's web browser isn't functional. We would like to add web browsing functionality so that OpenDevin can view documentation and other relevant information not stored in its parametric memory.

Playwright should be added to the backend to scrape web pages and send the following payload to the frontend on each page change:

{
  "url": "...",
  "screenshot": "..."
}

url is the current URL OpenDevin is viewing. screenshot is a base64-encoded screenshot of the page OpenDevin is viewing.

This allows the user to see what OpenDevin is viewing from the UI.

[boyuanzheng010]

We have implemented web browsing functionality with PlayWright in the SeeAct codebase. This codebase enable running web agents on live websites, including:

• Open a browser and go to the target websites
• Get HTML/CSS, screenshot image from the webpage
• Execute agent actions on webpage

I can adapt some functionality here to enable web browser.

[yimothysu]

@boyuanzheng010 That would be fantastic!

[nullnuller]

The latest pull has this error File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/nulled/Downloads/LLM_Applications/OpenDevin/opendevin/server/listen.py", line 1, in from opendevin.server.session import Session File "/home/nulled/Downloads/LLM_Applications/OpenDevin/opendevin/server/session.py", line 8, in from opendevin.action import ( File "/home/nulled/Downloads/LLM_Applications/OpenDevin/opendevin/action/init.py", line 3, in from .browse import BrowseURLAction File "/home/nulled/Downloads/LLM_Applications/OpenDevin/opendevin/action/browse.py", line 5, in from playwright.async_api import async_playwright ModuleNotFoundError: No module named 'playwright' make[1]: *** [Makefile:25: start-backend] Error 1 make[1]: Leaving directory '/home/nulled/Downloads/LLM_Applications/OpenDevin'

[yimothysu]

We need to add playwright to Pipenv.

[rbren]

Pretty sure this is good now