Add ability to save autofill data and trigger autofill on form elements

[yoavweiss]

Problem

Currently there is no way for developers to automate autofill in browsers and thus they can't test their HTML with the browser's autofill behaviour. Adding such an API would allow such testing and enable developers to write automated autofill tests (e.g. WPT).

Proposal

We are proposing to add 2 new end points to webdriver:

Endpoint /session/{session id}/autofill/save

This endpoint can be used to save an address in the autofill manager of the browser which is necessary to set up any automated tests. It should take a JSON object as input with the keys being the field name and the values being the actual value for that field.

Endpoint /session/{session id}/autofill/trigger

This endpoint will be used to trigger the autofill behaviour on a specific input field, similar to how a user would do it when using the browser manually. It should take the field's id attribute as parameter to specify which input field autofill should be triggered on.

[OrKoN]

@jgraham @whimboo @gsnedders WDYT?

[galich]

The idea seems valid. Let's discuss specifics:

1. There are a few ways "save" can be triggered: form submit event, page navigation, perhaps more? We could leverage "form submit" as a trigger to save.

2. There is "autofill" and "autofill", one is "automatically filling without user interaction" and another one is "user triggers autofill, see options, selects one and browser fills it". Do we want to have more precise test over how user triggers autofill vs what is suggested vs what is filled?

3. Are we interested in real life scenarios, like when fields have no id, or reuse id, formless or placed in multiple iframes? This might affect proposed API.

4. Are we talking about address autofill specifically or about more broad autofill (e.g. logins, credit cards)?

[theindra]

Thanks for all the great questions @galich! Here some thoughts...

1. Currently our intent was to use this to test the actual autofill behaviour (which depends on the html), and the save method would be just used to populate data in the browser's address store. For that purpose, we would suggest a simple "form submit" to trigger the save.
2. From my point of view, the UI flows you explained are part of the individuals browser's product philosophy. We are interested in testing HTML content in the context of autofill, hence the browser side is not the main focus from my perspective.
3. Yes, are absolutely interested in real world scenarios. We have a collection of actual test scenarios in this Draft PR. The points you raised regarding ID-less fields, reused IDs and fields across iframes are great, thanks! Our initial idea was to just use a field id of the element triggering the autofill. We forgot to add the id in our PR. How would you think we should change the proposed API's for those real world scenarios?
4. While our focus is mostly on addresses due to their discrepancies caused by the wide variety of address formats, I don't think the webdriver API would be limited to addresses only.

[galich]

1. Currently our intent was to use this to test the actual autofill behaviour (which depends on the html), and the save method would be just used to populate data in the browser's address store. For that purpose, we would suggest a simple "form submit" to trigger the save.

Form submit should work across all browsers, but what if browser asks a confirmation from user before saving data? We need some way to do "form submit with a flag to automatically accept saving an address". May be browsers could autosave when operating by web driver.

2. From my point of view, the UI flows you explained are part of the individuals browser's product philosophy. We are interested in testing HTML content in the context of autofill, hence the browser side is not the main focus from my perspective.

Sounds like "what was filled" part to me. I'm assuming we expect only 1 entry that can be filled or API would specify which address entry to fill?

3. Yes, are absolutely interested in real world scenarios. We have a collection of actual test scenarios in this [Draft PR](https://github.com/web-platform-tests/wpt/pull/44329/files). The points you raised regarding ID-less fields, reused IDs and fields across iframes are great, thanks!
   Our initial idea was to just use a field id of the element triggering the autofill. We forgot to add the id in our PR. How would you think we should change the proposed API's for those real world scenarios?

On top of my head, and @DimiDL can probably suggest more, we may want to specify from which iframe and which input the autofill is triggered. Also would be curious to see what happens when there are multiple forms across iframes with reused input names.

4. While our focus is mostly on addresses due to their discrepancies caused by the wide variety of address formats, I don't think the webdriver API would be limited to addresses only.

++ to making it not-address specific.

[battre]

With regard to filling: I am not particularly familiar with WebDriver but would it make sense to to pass a WebElement (API) for the trigger field instead of an id?

With regard to saving: I think that both approaches, 1) explicitly saving the values and 2) simulating a save that would be triggered by a form submission would be useful. I think that 1) would be simpler and possibly more inclusive (I don't know whether all user agents offer a save-on-submission feature). I would still like to see and could imagine a simple API that says "on next form submission save automatically".

[gsnedders]

With regard to filling: I am not particularly familiar with WebDriver but would it make sense to to pass a WebElement (API) for the trigger field instead of an id?

This is fundamentally what the design is; the ID is how Selenium's WebElement's get communicated over the wire. It's not what DOM considers IDs—it's a WebDriver specific ID.

[yoavweiss]

May be browsers could autosave when operating by web driver.

Yeah, I was thinking they'd autosave in testing scenarios.

Sounds like "what was filled" part to me. I'm assuming we expect only 1 entry that can be filled or API would specify which address entry to fill?

@OrKoN made some relevant suggestions on https://github.com/w3c/webdriver-bidi/pull/706#discussion_r1592961600

From my perspective a single entry would be fine, but I'm cool with expanding the flow to enable the test script to choose which.

On top of my head, and @DimiDL can probably suggest more, we may want to specify from which iframe and which input the autofill is triggered. Also would be curious to see what happens when there are multiple forms across iframes with reused input names

Again, @OrKoN made similar comments on https://github.com/w3c/webdriver-bidi/pull/706#discussion_r1592959545

Should we perhaps coalesce design efforts on one of these?