New selenium browser fails to retrieve data from for (example Twitter)

[kristoffernolgren]

Duplicates

• [X] I have searched the existing issues

Steps to reproduce 🕹

Try to use action browse_website to a page that is heavily reliant on javascript.

Current behavior 😯

Once auto-gpt tries to browse twitter accounts, the conclusions from auto-gpt can only detect

example: "Answer gathered from website: The text does not provide any information about recent tweets mentioning stocks. It is a message about enabling JavaScript to use Twitter.

Links: ['Help Center (https://help.twitter.com/using-twitter/twitter-supported-browsers)', 'Terms of Service (https://twitter.com/tos)', 'Privacy Policy (https://twitter.com/privacy)', 'Cookie Policy (https://support.twitter.com/articles/20170514)', 'Imprint (https://legal.twitter.com/imprint.html)']", <selenium.webdriver.chrome.webdriver.WebDriver (session="6ad00e143bf053610977dcee88cebaa5")>)

Expected behavior 🤔

Scraper manages JS and/or waits until site is fully loaded.

Your prompt 📝

Use the browse_website command to visit the Twitter accounts, the webpage to visit is https://twitter.com/[username]

[mflanagan1041]

twitter recently forced API payments and changed up who could access their developer API. unless selenium is browsing with your login to get those tweets you will not be able to get that data unless you have a valid Developer API account/key.

[kristoffernolgren]

For me, this happens for any site requesting cookies, inkl 4 out of 5 major news outlets tested, like cnbc, Reuter etc

[nileshtrivedi]

I would like to see a Javascript version of AutoGPT which will be able to make use of puppeteer-like scripts the way aicombinator does.

[iller75]

Got a similar issue, as soon as it tries to browse a website i often get this site provides no info about.... it only talks about cookies etc. And then it fails to read the info on the page.

[Bubble007]

similar problem

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions[bot]]

This issue was closed automatically because it has been stale for 10 days with no activity.