What happens if promises to Blobs are not resolved within a reasonable amount of time?

[snianu]

ClipboardItem has ClipboardItemDataType which is a Promise to Blob or DOMString. When web authors don't resolve these promises in a reasonable amount of time, what should the Browser do in that case? Also, what happens if the promises to Blobs are resolved after the Browser Window loses focus? or user switches to another tab (while the promises are still not resolved in the first tab), opens the same site and starts another copy? Should we add a check to verify if the Document is active or not after the promises are resolved and before we write the payload to the clipboard? @whsieh @mbrodesser @BoCupp-Microsoft @mkruisselbrink

[AshleyScirra]

The spec's user activation rules solve this: you get 1 second. If the clipboard promises only have 1 second, then there's no need for the promise API at all, and it can all just be done under user activation rules.

[mbrodesser]

The spec's user activation rules solve this: you get 1 second. If the clipboard promises only have 1 second, then there's no need for the promise API at all, and it can all just be done under user activation rules.

Thanks for bringing up user activation.

User activation is required when calling clipboard.write() or clipboard.writeText(), see https://w3c.github.io/clipboard-apis/#dom-clipboard-writetext down to https://html.spec.whatwg.org/multipage/interaction.html#transient-activation-duration. The spec only mentions that the duration should be "at most a few seconds", but even if it were more precise, it wouldn't answer the questions raised in https://github.com/w3c/clipboard-apis/issues/161#issue-1037708147.

@snianu: thanks for bringing up the questions, I didn't investigate them exhaustively yet, but AFAIU, the behavior is currently not covered by the spec. But it should be. CC @annevk

[mbrodesser]

Thanks for raising those questions. Very important.

ClipboardItem has ClipboardItemDataType which is a Promise to Blob or DOMString. When web authors don't resolve these promises in a reasonable amount of time, what should the Browser do in that case?

It seems reasonable to consider relevant use cases to determine how browsers should behave in that case. Perhaps it's reasonable to assume that web-devs either

1. write small amounts of data to the clipboard, so that they can resolve those promises in "a reasonable amount of time". A few seconds seems reasonable.
2. write large amounts of data to the clipboard, which they can prepare before calling clipboard.write.

However, I don't know if 2) is true in practice. @AshleyScirra: since you seem to be a web-dev, if you could contribute relevant use cases, that'd be helpful.

Also, what happens if the promises to Blobs are resolved after the Browser Window loses focus?

Interesting question. One reason to use the async clipboard API is that it doesn't block (mentioned e.g. at https://www.tiny.cloud/blog/new-async-clipboard-api/). So that could mean, a use case for end users could be to trigger a write(), but knowing that it may take time, switch to another application. So for that use-case it would be annoying to cancel the write().

or user switches to another tab (while the promises are still not resolved in the first tab), opens the same site and starts another copy? Should we add a check to verify if the Document is active or not after the promises are resolved and before we write the payload to the clipboard? @whsieh @mbrodesser @BoCupp-Microsoft @mkruisselbrink

Perhaps it's reasonable to start with a short timeout, e.g. a few seconds, and not checking if the document is active. If, in practice it turns out to be an issue, one could still consider more subtle behavior and specification.

[tomayac]

I’m working on an application that works with potentially huge SVGs that get lazily optimized with svgo before being put on the clipboard (or being saved to a file). So far I have worked with an ugly but functional solution you can see in https://bugs.webkit.org/show_bug.cgi?id=222262#c5. I haven’t run into it yet, but in the worst case my svgo optimization task takes longer enough for the user gesture to be expired. Not sure if this is a use case you were looking for, but thought I’d share it.

Cheers, Tom

On Mon 1. Nov 2021 at 12:43 mbrodesser @.***> wrote:

Thanks for raising those questions. Very important.

ClipboardItem has ClipboardItemDataType which is a Promise to Blob or DOMString. When web authors don't resolve these promises in a reasonable amount of time, what should the Browser do in that case?

It seems reasonable to consider relevant use cases to determine how browsers should behave in that case. Perhaps it's reasonable to assume that web-devs either

1. write small amounts of data to the clipboard, so that they can resolve those promises in "a reasonable amount of time". A few seconds seems reasonable.
2. write large amounts of data to the clipboard, which they can prepare before calling clipboard.write.

However, I don't know if 2) is true in practice. @AshleyScirra https://github.com/AshleyScirra: since you seem to be a web-dev, if you could contribute relevant use cases, that'd be helpful.

Also, what happens if the promises to Blobs are resolved after the Browser Window loses focus?

Interesting question. One reason to use the async clipboard API is that it doesn't block (mentioned e.g. at https://www.tiny.cloud/blog/new-async-clipboard-api/). So that could mean, a use case for end users could be to trigger a write(), but knowing that it may take time, switch to another application. So for that use-case it would be annoying to cancel the write().

or user switches to another tab (while the promises are still not resolved in the first tab), opens the same site and starts another copy? Should we add a check to verify if the Document is active or not after the promises are resolved and before we write the payload to the clipboard? @whsieh https://github.com/whsieh @mbrodesser https://github.com/mbrodesser @BoCupp-Microsoft https://github.com/BoCupp-Microsoft @mkruisselbrink https://github.com/mkruisselbrink

Perhaps it's reasonable to start with a short timeout, e.g. a few seconds, and not checking if the document is active. If, in practice it turns out to be an issue, one could still consider more subtle behavior and specification.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/w3c/clipboard-apis/issues/161#issuecomment-956163499, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDSDFG4KFBD4LWMDKNEYLUJZ4NLANCNFSM5G26TKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thomas Steiner, PhD—Developer Advocate (https://blog.tomayac.com, https://twitter.com/tomayac)

Google Germany GmbH, ABC-Str. 19, 20354 Hamburg, Germany Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.3.2 (GNU/Linux)

iFy0uwAntT0bE3xtRa5AfeCheCkthAtTh3reSabiGbl0ck0fjumBl3DCharaCTersAttH3b0ttom. hTtPs://xKcd.cOm/1181/ -----END PGP SIGNATURE-----

[AshleyScirra]

In our case we just need to read a blob. Usually it will take a negligible amount of time. However since reading a blob is async it means once the result is available, a following clipboard call is not synchronously inside a user gesture event. In Chrome this is not a problem as the user activation rules allow a short timeout. In other browsers that do not implement these user activation rules, it's a problem, but that problem can be solved by implementing Chrome's user activation rules.

[mbrodesser]

@tomayac: thanks for your feedback.

I’m working on an application that works with potentially huge SVGs that get lazily optimized with svgo before being put on the clipboard (or being saved to a file). So far I have worked with an ugly but functional solution you can see in https://bugs.webkit.org/show_bug.cgi?id=222262#c5. I haven’t run into it yet, but in the worst case my svgo optimization task takes longer enough for the user gesture to be expired. Not sure if this is a use case you were looking for, but thought I’d share it. Cheers, Tom

Yes, it was one of the use cases I was looking for. However, it still remains unclear, whether reaching that worst case is likely to happen in practice, or whether it's an artificial scenario. :-) Any clue about that?

[mbrodesser]

Thanks for the feedback.

In our case we just need to read a blob. Usually it will take a negligible amount of time. However since reading a blob is async it means once the result is available, a following clipboard call is not synchronously inside a user gesture event. In Chrome this is not a problem as the user activation rules allow a short timeout. In other browsers that do not implement these user activation rules, it's a problem, but that problem can be solved by implementing Chrome's user activation rules.

According to https://bugs.webkit.org/show_bug.cgi?id=222262#c3 there's a workaround not requiring a different user activation API. In order to solve the reporter's issue, it seems reasonable to not extend the scope to different user activation APIs, unless required.

[snianu]

@mbrodesser

The spec only mentions that the duration should be "at most a few seconds", but even if it were more precise, it wouldn't answer the questions raised in #161 (comment).

I agree with this. User activation is probably orthogonal to this discussion, but since this was also brought in an Intent-to-Ship thread, I'd like to share some thoughts:

Currently there is no user activation requirements in async clipboard read/write in Chromium browsers. e.g. https://clumsy-garnet-meeting.glitch.me/. Just load this page and refresh it without clicking anywhere on the page, you will see the content will get written to the clipboard without any gesture. Safari, due to security concerns, implemented a gesture requirement to access clipboard via async clipboard APIs. The read()/write() method can only be called inside a trusted user gesture event handler, but the promises to Blobs can be resolved later which gives the web authors the flexibility to not block the UI thread to populate the payload.

There is also a proposal to add Pickling API to the async clipboard API that enables web authors to read/write unsanitized content. Initially we added a transient user activation requirement because the API lets web authors read/write unsanitized content using a custom clipboard format. Both Chrome security team(see "User Gesture Requirement" section in https://github.com/w3c/editing/issues/315) and TAG raised the concern that transient user activation is NOT sufficient to give clipboard access to web authors.

The promises to Blobs in the ClipboardItem is required to not only support the Pickling API's user gesture event handling requirements, but also improve the security by adding an even stronger signal of the user intent to copy. We also talked about this in the Editing WG call with Apple and they are opposed to transient user activation and I believe Firefox raised concerns as well(https://github.com/w3c/clipboard-apis/issues/52#issuecomment-599443986).

[tomayac]

Yes, it was one of the use cases I was looking for. However, it still remains unclear, whether reaching that worst case is likely to happen in practice, or whether it's an artificial scenario. :-) Any clue about that?

Personally I haven't, but @inexorabletash has mentioned running into this (internal email thread). I noticed an issue with the following sequence, though:

1. Start the copy operation.
2. Wait for the data to be ready.
3. Switch to different tab while the data is being prepared.
4. The tab in the background starts the actual copy operation when the data is ready.
5. The operation throws since the tab is not focused.

[snianu]

@tomayac The tab focus is a bit complicated issue. There could also be two tabs racing to write content to the clipboard if the user (for some weird reason) switches back and forth in quick succession(using ctrl+tab & ctrl+shift+tab) and initiates copy from both tabs. I discussed this with @BoCupp-Microsoft and we think we need a mechanism to store the ClipboardItem somewhere in the Document, and based on which tab has focus, we will write the ClipboardItem fetched from the focused Document to the clipboard. The list will also be helpful if the web authors write multiple ClipboardItems in a loop (I know this is a contrived scenario, but just wanted to mention it for completeness). The last ClipboardItem that was set by the web author before the write to the clipboard is initiated, wins. Please let us know what you think as we need more feedback from web authors to come up with a valid algorithm.

[mbrodesser]

Thanks for the response, @snianu.

Safari, due to security concerns, implemented a gesture requirement to access clipboard via async clipboard APIs. The read()/write() method can only be called inside a trusted user gesture event handler, but the promises to Blobs can be resolved later which gives the web authors the flexibility to not block the UI thread to populate the payload.

If this is correct, then I wonder if https://github.com/w3c/clipboard-apis/issues/161#issuecomment-956168974 is indeed an issue.

[snianu]

@mbrodesser Yep, that would definitely be an issue. I think we could relax the user gesture requirement to only have transient user activation during read/write call (along with permissions for the read call, if the browser has implemented clipboard permissions). That way authors have some time to at least initiate the read/write call, but resolving the promises to Blobs doesn't need to have this restriction, nor does writing to the system clipboard. Thoughts?

[mbrodesser]

@mbrodesser Yep, that would definitely be an issue.

Thanks, thought about the issue again and agree.

I think we could relax the user gesture requirement to only have transient user activation during read/write call (along with permissions for the read call, if the browser has implemented clipboard permissions). That way authors have some time to at least initiate the read/write call, but resolving the promises to Blobs doesn't need to have this restriction, nor does writing to the system clipboard. Thoughts?

@snianu: this could be confusing to users, e.g. when they switch tabs. Or when malicious websites postpone resolving those promises for a long time, e.g. minutes.

I wonder if this only affects write() and writeText(), but neither read() nor readText(). If not, that may simplify thinking about this.

It seems worth considering the end user perspective on this, on Safari (for Gecko/Firefox we'll likely implement the same). What is the UX there, currently? Is a "Copy" button shown? I'm only aware of the read*() case, which shows a "Paste" button. CC @rniwa

I think end users need to be informed visually when a write*() call starts, as long as it hasn't finished and when it's indeed finished. Browsers (Safari and Firefox, at least) should ensure this; it shouldn't be up to the discretion of web-devs to ensure this. If we can agree on this, that may also help to specify tab switching cases properly. My gut feeling is that writing large amounts of data to the clipboard happens rarely, and it may be acceptable for end users to not switch tabs, as long as the write-process is ongoing. It seems worth keeping this as simple as possible. Those are just thoughts, my mental model of this is still growing.

[mbrodesser]

@tomayac The tab focus is a bit complicated issue. There could also be two tabs racing to write content to the clipboard if the user (for some weird reason) switches back and forth in quick succession(using ctrl+tab & ctrl+shift+tab) and initiates copy from both tabs. I discussed this with @BoCupp-Microsoft and we think we need a mechanism to store the ClipboardItem somewhere in the Document, and based on which tab has focus, we will write the ClipboardItem fetched from the focused Document to the clipboard. The list will also be helpful if the web authors write multiple ClipboardItems in a loop (I know this is a contrived scenario, but just wanted to mention it for completeness). The last ClipboardItem that was set by the web author before the write to the clipboard is initiated, wins. Please let us know what you think as we need more feedback from web authors to come up with a valid algorithm.

That proposal seems confusing for end users. I think they should be informed clearly when a write operation starts, is going on and has ended. Two write operations should not be allowed to overlap. If there's indeed a time intensive write*() going on and an end user wants to start a different one, the first one should clearly be aborted.

[AshleyScirra]

I think a deferral style API for the web platform would considerably simplify this, and some other related problems, without having to invent API-specific solutions. For example:

document.addEventListener("click", e =>
{
	// user gesture rules allow certain APIs synchronously here
	const deferral = e.getDeferral();
	
	prepareSomeClipboardDataAsync().then(data =>
	{
		// user gesture rules now apply synchronously here
		deferral.use();
		
		// can now write to clipboard normally
		clipboard.write(data);
	});
});

The core problem is certain API calls like clipboard.write() are only allowed at certain times, and doing async work means it might not be allowed by the time the async work finishes. A deferral system allows the developer to save the allowed call as a deferral, and then call it at a time of their choosing later on. This avoids the need to have a special promise mechanism specific to the clipboard API, and generalises the solution to work with any data processing with any API.

To ensure single-use, getDeferral() would only work if no sensitive APIs were previously called, and subsequently no sensitive APIs can be used synchronously either, until deferral.use() is called. That itself is also single use and only allows a single use of a sensitive API synchronously after the call. In this case the app now provides all the clipboard data up-front so the browser can validate it or prompt for it based on the initial call. The browser can also impose further restrictions on deferral.use(), such as imposing a timeout, throwing if the document is not visible, and so on.

[mbrodesser]

I think a deferral style API for the web platform would considerably simplify this, and some other related problems, without having to invent API-specific solutions.

Thanks for bringing up the idea. I'm not familiar with the details of deferrals, and the exact reasons why they're currently not supported in JS, but AFAIU, they're undesirable, because they complicate code. See https://stackoverflow.com/a/34971125.

[mbrodesser]

@snianu

Another, seemingly general problem: if script is allowed to write to the clipboard for longer than the transient user activation duration (= a few seconds), should there be another timeout value? If so, what would it be?

The answer is related to the question of how heavy such write operations should be allowed to be. Edit: perhaps it could make sense to design write*() only for non-heavy write operations. Separating any computational work (like applying svgo, mentioned above) from actually writing the produced work to the clipboard might require more clicks from an end user, but it'd be clearer. That might resolve this hole issue.

[AshleyScirra]

@mbrodesser - that link appears to refer to something different that happens to share the name "defer". I don't think it's relevant to what I proposed.

[mbrodesser]

@mbrodesser - that link appears to refer to something different that happens to share the name "defer". I don't think it's relevant to what I proposed.

IIUIC, it refers to the same concept: deferrals.

[snianu]

@mbrodesser Thank you for your response! We are planning to discuss this in the Editing WG call on 11/12, but here are some thoughts:

I think end users need to be informed visually when a write*() call starts, as long as it hasn't finished and when it's indeed finished.

Yes, this is what we are planning to do in Office online products (at least Excel online for now). Since the payload is big, it takes an arbitrary amount of time to resolve the promises, so we are going to show a message or something if the user tries to move the focus away from the tab.

In general, I think setting any time limit would be problematic because you could always create something that would break it.

Two write operations should not be allowed to overlap. If there's indeed a time intensive write*() going on and an end user wants to start a different one, the first one should clearly be aborted.

Yes, I agree that there shouldn't be any overlap, but from Chromium's perspective, it's hard to tell which one to cancel as the write takes place in the browser process and there could be many renderer processes initiating the write call at different intervals (due to payload size etc).

One way to solve all these issues related to timing and stuff is to implement Delay Rendering that gives authors the ability to serialize only the payload associated with the MIME type requested by the OS, but not sure when we are going to work on it (hopefully soon!).

[mbrodesser]

@mbrodesser Thank you for your response! We are planning to discuss this in the Editing WG call on 11/12, but here are some thoughts:

Thanks for sharing them.

I think end users need to be informed visually when a write*() call starts, as long as it hasn't finished and when it's indeed finished.

Yes, this is what we are planning to do in Office online products (at least Excel online for now). Since the payload is big, it takes an arbitrary amount of time to resolve the promises, so we are going to show a message or something if the user tries to move the focus away from the tab.

In general, I think setting any time limit would be problematic because you could always create something that would break it.

True.

Two write operations should not be allowed to overlap. If there's indeed a time intensive write*() going on and an end user wants to start a different one, the first one should clearly be aborted.

Yes, I agree that there shouldn't be any overlap,

Just want to stress this is important. For end users' understanding and to keep the clipboard API simple. If a write*() call immediately fails, when another write*() call is still running, no overlap would be possible. Moreover, if only the active document would be allowed to write*(), one tab / iframe / website couldn't block other tabs from writing.

but from Chromium's perspective, it's hard to tell which one to cancel as the write takes place in the browser process and there could be many renderer processes initiating the write call at different intervals (due to payload size etc).

One way to solve all these issues related to timing and stuff is to implement Delay Rendering that gives authors the ability to serialize only the payload associated with the MIME type requested by the OS, but not sure when we are going to work on it (hopefully soon!).

IIUC, this wouldn't solve the timing problem. The payload associated with one MIME type could still be large.

[css-meeting-bot]

The Web Editing Working Group just discussed What happens if promises to Blobs are not resolved within a reasonable amount of time? (#161).

The full IRC log of that discussion

<Travis> topic: What happens if promises to Blobs are not resolved within a reasonable amount of time? (#161)
<Travis> BoCupp: snianu added promises to dom strings or blobs so that clipboard could take a promise to its data in a write.
<Travis> .. we were hearing feedback about the open-ended-ness of the promise? How long do we wait?
<Travis> .. 2 seconds? a minute? indefinitely?
<Travis> .. other issues: what if a second write happens? Is the first canceled? What about a write in another tab (across origin)?
<Travis> .. is it what's in the foreground that takes precedence?
<Travis> .. is it a matter of luck?
<Travis> .. lots of corner cases!
<Travis> whsieh: this is shipping.
<Travis> .. out behavior is: no hard-coded timeout...
<Travis> .. but if you leave the tab, the promise (prior call) is rejected.
<Travis> BoCupp: If the tab leaves the foreground, or a second write occurs in the same time, this rejects the previous write promise.
<Travis> whsieh: yes.
<Travis> .. furthermore, you need to be the foreground tab in order to use this; otherwise it rejects immedatiatly due to no user ineraction.
<Travis> .. if you switch away, then use another site to try to write to the clipboard; the subsequent write would kill the original.
<Travis> BoCupp: We had one mild concern.. fast-finger copy-tab-switch-paste...
<Travis> .. if this switch that cancels isn't fast enough...
<Travis> whsieh: yeaaaaah, I see that. In practice, not a concern.
<Travis> BoCupp: I like the rules you noted whsieh
<Travis> snianu: maybe one issue with checking active document. In chromium, browser process shows a dialog. When you click OK the promise resolves.
<Travis> .. between the time the OK is clicked, then the promise resolves, and inbetween that time the document loses focus. If the check for an active document occurs during that time, there could be a race.
<Travis> .. might be a chromium-specific case.
<Travis> BoCupp: for our permission dialog, I think we could ensure the scenario works.
<Travis> .. if it was other dialogs (unbounded) that might be a problem, but might be an edge case.
<Travis> github: https://github.com/w3c/clipboard-apis/issues/161
<Travis> BoCupp: I think we can make it work for this case.

[snianu]

Since we have come to an agreement (mentioned in the meeting notes) that Safari's behavior makes sense and is aligned with our proposal, we are going to make that change in Chromium as well. I'm going to add this to the spec. Keeping the issue open for any feedback.

[mbrodesser]

Since we have come to an agreement (mentioned in the meeting notes) that Safari's behavior makes sense and is aligned with our proposal, we are going to make that change in Chromium as well. I'm going to add this to the spec. Keeping the issue open for any feedback.

@snianu Thanks for the update, including the text from the meeting bot. I've tried to finding the whole meeting minutes but didn't succeed. If already available in the web, can someone please them here, or, preferably, explain a future-proof way to find such notes.

[snianu]

The minutes are recorded in the Editing WG meeting notes. You probably have to subscribe to the group to receive regular updates.

[BoCupp-Microsoft]

@snianu could you add a link here to the PR that will close this issue?

[snianu]

Here is the PR that has the promise related changes along with all the other changes related to async clipboard API.

[annevk]

Let's keep this open and close it when the PR lands (by writing "closes #..." in the commit message).

[saschanaz]

Did #158 include the fix or not?