Handle autofill-like inputs with dynamic autocomplete

[suchintan]

Skyvern fails to fill out autocomplete fields

https://discordapp.com/channels/1212486326352617534/1214296823066534021/1218050218856153118

https://github.com/Skyvern-AI/skyvern/assets/3853670/52820c83-2870-44c9-a583-c6213d067a91

[AlexAndrei98]

👀

[wintonzheng]

Related Context:

We have some basic js call detection here in the agent loop: https://github.com/Skyvern-AI/skyvern/blob/ff4be0de9ef79a384625fd74ba79af4e2fb41539/skyvern/forge/agent.py#L431. Once we detect js call / event, we stop executing the remaining actions and ask skyvern to reprocess the DOM.

Proposal

similarly for autofill, we need to add code to detect autocompletion (by detecting js events or DOM updates) and pass information to the agent to stop handling the remaining actions if any. Then skyvern will be able to reprocess the DOM, plan the actions based on the autocompleted suggestion elements (pick the correct suggestion to click)

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.