[Feature]: Enable storage partitioning and consider expanding storage state API to support storage keys

[recvfrom]

🚀 Feature Request

Storage partitioning is now enabled by default in all major browsers (ex) but seems to be disabled for browsers by playwright:

• Chrome: the ThirdPartyStoragePartitioning feature flag is disabled
• Firefox: network.cookie.cookieBehavior is not set to 5
• Webkit: referenced here

For Firefox this was previously tracked by #31275 but ultimately closed.

For Chromium, we are planning to remove the ThirdPartyStoragePartitioning feature flag, so there will not be a way to opt out of storage partitioning. That is tracked by https://crbug.com/410491202

Partitioning seems to have been disabled due to #32230. The issue from that bug is that with the following code, when third-party storage partitioning is enabled, the localStorage is not set in the third-party iframe:

  const context = await browser.newContext({
    storageState: {
      cookies: [],
      origins: [{ origin: 'https://clausa.app.carto.com', localStorage: [{ name: 'hello', value: 'world' }]}]
    }
  })

Given that the storageState dictionary only has keys corresponding to origins, with storage partitioning enabled it makes sense that the storage partition chosen to write this data into is the first-party context corresponding to that origin (from the code in the example above, the first-party storage partition would be top-level site: https://carto.com, origin: https://clausa.app.carto.com, has-cross-site-ancestor: false). It's WAI that that local storage data would not be accessible from a third-party context when storage partitioning is enabled, since the top-level site and has-cross-site-ancestor fields of the storage key would be different.

If there was enough interest in setting state in specific third-party contexts, storageState could be extended so that storage keys can be specified in addition to just origins. I'm not sure how playwright sets state under the hood, but this should at least be possible for Chrome. What fields make up a storage key is slightly different across browsers today, but a standard format has been proposed and has consensus IIUC: https://github.com/whatwg/storage/pull/182

Example

No response

Motivation

Testing what browsers are actually shipping would be beneficial to playwright users.

Also, once the ThirdPartyStoragePartitioning feature flag is removed, storage partitioning will be enabled for Chrome in playwright but not for other browsers. It'd be better if the feature was enabled proactively across all of the browsers together.

[dgozman]

@recvfrom Thank you for reaching out!

We would love to turn this feature on. As you pointed out, our current capabilities of saving/restoring the storage are not exactly compatible with partitioning, and #32230 shows that it is sometimes important for storage to be available in the cross-process iframe.

What do you think about exposing two CDP methods like Storage.getStorage/setStorage? If that's available, we can rely on Chrome recognizing the saved storage format.

Without CDP support, it does not seem practical to replicate all the intricate details of storage partitioning outside of the browser, so disabling the feature is the only way to make things work for now.

[recvfrom]

Thanks @dgozman, the DevTools team did a lot of work to support storage partitioning as well, including making it so that the Application tab broadly supports displaying / setting state from/for third-party contexts. It looks like that uses DOMStorage.getDOMStorageItems and DOMStorage.setDOMStorageItem, with an example request for the latter looking like:

{"storageId":{"storageKey":"https://xchrdw.github.io/^0https://xchrdw2.github.io","isLocalStorage":true},"key":"asdf","value":"asdf"}

Regarding how to actually get the value for the storageKey parameter, DevTools uses Storage.getStorageKey where a request looks like {"frameId":"9C4502664F14598512F740ECAF12CD5B"} and the response looks like {"storageKey":"https://xchrdw.github.io/^0https://xchrdw2.github.io"}. That's useful if there's already a frame with the storage key that you want to set storage in, but less so if there isn't.

For making the newContext storageState parameter be able to set storage in arbitrary third-party contexts, would you need a CDP method that takes in a top-level site string, an origin string, and has-cross-site-ancestor boolean and provides the storageKey value to use for the other APIs? Or did you have something else in mind?

[dgozman]

Thanks @dgozman, the DevTools team did a lot of work to support storage partitioning as well, including making it so that the Application tab broadly supports displaying / setting state from/for third-party contexts. It looks like that uses DOMStorage.getDOMStorageItems and DOMStorage.setDOMStorageItem, with an example request for the latter looking like:

{"storageId":{"storageKey":"https://xchrdw.github.io/^0https://xchrdw2.github.io","isLocalStorage":true},"key":"asdf","value":"asdf"}

That's great to know, I was not aware of it. Thank you for pointing it out!

Regarding how to actually get the value for the storageKey parameter, DevTools uses Storage.getStorageKey where a request looks like {"frameId":"9C4502664F14598512F740ECAF12CD5B"} and the response looks like {"storageKey":"https://xchrdw.github.io/^0https://xchrdw2.github.io"}. That's useful if there's already a frame with the storage key that you want to set storage in, but less so if there isn't.

Yeah, this does not work if I'd like to save storage after the fact, for all iframes that have been on the page(s). I see that DOMStorage.getDOMStorageItems is implemented in blink. I think we would need something in the browser around StoragePartitionImpl to be able to read all storage in bulk, for all storage keys available in the storage, corresponding to all pages/iframes visited in the BrowserContext. Similar to Network.getCookies.

For making the newContext storageState parameter be able to set storage in arbitrary third-party contexts, would you need a CDP method that takes in a top-level site string, an origin string, and has-cross-site-ancestor boolean and provides the storageKey value to use for the other APIs? Or did you have something else in mind?

I expect such a solution to not work, because renderer won't have proper permissions to set storage for an unrelated storageKey. So we would need to iterate over various origins, create a new renderer process for each, and then it could work. I'd much prefer a "bulk set storage" method in the browser process, similar to Network.setCookies.

[recvfrom]

Thanks for the additional context!

I think we would need something in the browser around StoragePartitionImpl to be able to read all storage in bulk, for all storage keys available in the storage, corresponding to all pages/iframes visited in the BrowserContext. Similar to Network.getCookies.

Instead of Network.getCookies I'm assuming you mean Storage.getCookies that takes the Browser.BrowserContextID as a parameter? That makes sense... I'll open a feature request for this.

I expect such a solution to not work, because renderer won't have proper permissions to set storage for an unrelated storageKey. So we would need to iterate over various origins, create a new renderer process for each, and then it could work. I'd much prefer a "bulk set storage" method in the browser process, similar to Network.setCookies.

Oh interesting... The example I gave in the description is setting cross-origin data. Does playwright have to iterate over origins, create a new renderer process for each, and set the storage using that today?

[recvfrom]

I'll open a feature request for this.

I've opened https://crbug.com/468317746