eyedropper-api

eyedropper-api copied to clipboard

Hi there!

Chromium Web Platform security reviewer here. It appears to us that this API allows an origin to learn the color of cross-origin pixels, in violation of the same-origin policy. This is not unlike the getCurrentDisplayMedia() API which allows capturing a tab as a video feed.

One important mitigating factor is that this API allows access to a single pixel, i.e. a few bits at most. In addition, the flow is explicitly mediated by the user, which must affirmatively decide to pick a pixel.

Still, a somewhat-contrived yet effective attack would be the following:

• attacker.example wants to learn whether or not the user is signed in to bank.example
• attacker.example embeds a hundred iframes pointing to bank.example, tiling the entire viewport
  • each iframe is scrolled to the location of a "sign in/out" button whose color changes with sign-in status
  • each iframe is additionally made mostly-transparent, such that the result is hard to notice by the user
• attacker.example then coaches the user into clicking somewhere, and opens the eyedropper
• the user clicks anywhere on the page, perhaps as a reflex given the unexpected cursor change, or is coached into doing so
• attacker.com learns the color of the button, and the "signed in" bit it was after

This should be called out in the specification and explainer's security considerations, and the S&P self-review.

Beyond that, what to do? There is a whole spectrum of possible mitigations which we are keen to explore with you.

At the strictest, in order to uphold the SOP, we could consider requiring that contexts calling the Eyedropper API be crossOriginIsolated + an affirmative opt-in (in the form of a header, such as considered for getViewportMedia()) from embedded iframes signalling that they are fine with being read by their embedder. If the applications you are targeting do not need to embed much third-party content, this might be the best solution.

At the least, we can consider some mitigations to both reduce the success rate of such attacks and their payoff:

• consider showing a "do you want to allow this web page to keep creating eyedroppers?" prompt after a few calls to Eyedropper.open() have been dismissed with ESC
• on the first call to Eyedropper.open() on a document, show a message explaining the modal shift (and how to bail out) and prevent interactions for say 1 second
• rate-limit calls to Eyedropper.open() to some reasonable value (every 5s? using token bucket?)
• or any other mechanism you feel would do a better job!

If such an approach is chosen, care should be taken to monitor the use of this API in the wild for abuse after it ships. Though that is hardly fool-proof, it would inform whether follow-up interventions are needed.

cc @arturjanc

cc @engedy

Keep in mind that this API isn't limited to reading pixels from the current browser window; it can read any pixel on the user's screen. As such, features like crossOriginIsolated aren't applicable against some threats, although obviously a malicious page does tend to have more control over the UI positioning of web content.

That's true! crossOriginIsolated has no bearing on threats outside the web content area.

What I find somewhat unintuitive about these APIs is precisely that the most dangerous surface to be sharing is not the desktop, or other windows outside Chrome's control, but the web content area under the control of the page calling the API. This because the page knows what is being shared and can embed particularly sensitive content.

Any news on this front?

There's some discussion of this threat vector and its design mitigations in the original explainer: https://github.com/MicrosoftEdge/MSEdgeExplainers/issues/385#issuecomment-761220244

Thanks for the pointer! It seems that the crossOriginIsolated + opt-in solution was already raised at the time. That would be the cleanest mitigation indeed. I'm curious to hear about the spec editors' views on this issue?

In my point of view, I feel

on the first call to Eyedropper.open() on a document, show a message explaining the modal shift (and how to bail out) and prevent interactions for say 1 second

probably makes most of the sense among others. This is at the end a low-traffic channel for leaking information.

@letitz - thanks for the issue! MicrosoftEdge/MSEdgeExplainers#385 is a discussion about a more powerful version of the API, we have decided to not allow multi-color selection and not provide the color selection coordinates anymore.

there are some mitigations that are currently in place:

• require consumable user activation to open the eyedropper
• require user action to select a color
• provide a way for the user to exit the eyedropper without selecting a color and not allow the behavior to be canceled by the author
• prevent color selection for a period after it was opened to ensure that the user had a chance to see the UI

I agree that is important to monitor the use of this API in the wild for abuse to see if any follow-up mitigations are needed.

Thanks for the replies! I am glad to hear about the existing mitigations.

This should be called out in the specification and explainer's security considerations, and the S&P self-review.

I think this point still stands. The explainer mentions privacy but not security explicitly, and nowhere does it mention the SOP. The spec also does not call out the risk of violating the SOP, focusing more on the risk that websites would learn about content outside the browser window. As I argued in my initial comment, I think this glosses over the greatest risk posed by the API.

• prevent color selection for a period after it was opened to ensure that the user had a chance to see the UI

That's great! It is exactly one of the mitigations I was asking about in my initial comment. As far as I can tell, however, this is only mentioned as a "should" in the non-normative "Security and Privacy considerations" section of the spec. Can it be upgraded to a normative requirement, and mentioned in the explainer?

In addition, I think the API would still benefit from a way to prevent abusive pages from constantly creating eyedroppers until the user mistakenly clicks somewhere interesting:

• consider showing a "do you want to allow this web page to keep creating eyedroppers?" prompt after a few calls to Eyedropper.open() have been dismissed with ESC
• rate-limit calls to Eyedropper.open() to some reasonable value (every 5s? using token bucket?)

What do you think of those options? I am open to other suggestions as well.

I think the current (lack of) UI also assumes that an eyedropper is something users are familiar with, which I would not expect to be the case.

• consider showing a "do you want to allow this web page to keep creating eyedroppers?" prompt after a few calls to Eyedropper.open() have been dismissed with ESC

I think that this may be something worth exploring, but we should consider that even now since the eyedropper requires consumable user activation the user would have to click each time on the page to have it appear. If the page is abusive, the user still has the option to dismiss the eyedropper via ESC or close the page (if it is constantly creating an eyedropper after clicking anywhere).

• rate-limit calls to Eyedropper.open() to some reasonable value (every 5s? using token bucket?)

This approach would break some usage scenarios (e.g. the user should be able to quickly reopen the eyedropper to edit a previous wrong selection).

I think the current (lack of) UI also assumes that an eyedropper is something users are familiar with, which I would not expect to be the case.

Chromium has provided an eyedropper with a similar UX via input type="color" for a while. I haven't seen any feedback that additional usage information would be needed.

I think that this may be something worth exploring, but we should consider that even now since the eyedropper requires consumable user activation the user would have to click each time on the page to have it appear. If the page is abusive, the user still has the option to dismiss the eyedropper via ESC or close the page (if it is constantly creating an eyedropper after clicking anywhere).

Understood. I worry about the possibility that the user might end up inadvertently clicking somewhere/anywhere (and anywhere might be enough for an attacker) if the website is persistent enough, instead of ESC and before deciding to close the page. As an attacker, I would simply try again and again until I win, to improve my odds overall.

With <input type=color> it's extremely clear from context what is going on. That's completely different from your cursor suddenly changing into a picker as you go about your day. And the cursor might already look funky with cursor:url() which makes it even more likely you'd miss it.

FWIW I'd be somewhat uncomfortable with a mitigation of "let's monitor usage and make sure the feature is not abused after launch", even on top of the current mitigations mentioned above. For a feature that can reveal cross-origin data, it seems prudent to have a more principled approach.

Short of requiring a cross-origin isolated context (which would be the safe solution here), I'm wondering if the browser could track the provenance of the sampled pixel and ask for permission if the requester would not normally have the ability to read it (e.g. if the source is cross-origin). This could be a one-time prompt so that applications that will use the Eye Droppper API a lot (and expect it to be used on cross-origin content) could rely on this permission. Ideally, this would be transparent to the application so that it could use the API without permission for same-origin content, the prompt would only show up if the user selects a cross-origin pixel. Depending on how we expect the API to be used, this may end up resulting in better UX than applying a combination of less robust mitigations.

Maybe it can just require the cursor change to stay for longer, and require the cursor to move further before the color is picked? For example, if the user clicks within 1s the color is not picked (and it keeps in the eyedropper state), and if the user clicks without the cursor leaving, say 10px radius from where it starts, it counts as a cancel. It would be very weird that a user wants to pick the color from where the eyedropper starts, or is able to pick the color in a very short time, so maybe something like these could be a reasonable mitigation without involving more restrictions on legitimate use cases?

For a feature that can reveal cross-origin data, it seems prudent to have a more principled approach.

Note that input type="color" that provides eyedropper functionality has been available for a while and there have been no reports of it being abused.

The lack of reported abuse is generally not a sufficient condition for an API to be safe with respect to the web security model.

Are you aware of a discussion about the risks of cross-origin information disclosure for the input type="color" feature? If the same concerns have been raised and resolved, I agree it would be a useful reference here; otherwise, there's a possibility that this issue was missed in the reviews for that feature.

Another thing to keep in mind that the EyeDropper API is supposed to have a different UX from the input#type=color eyedropper one in ways that @annevk mentioned above, which does affect the security properties of the feature.

Re: mitigations, it seems hard to find a set of rules that will eliminate the potential for information leaks. For example, it's possible craft a completely legitimately looking image composed entirely of cross-origin resources; any pixel that the user is enticed to select may reveal cross-origin information in ways that will be difficult to address by tweaking the EyeDropper UX. That's why my guess is that it's important to figure out a shape for this feature that fully resolves this concern, rather than focus on necessarily imperfect mitigations.

For example, it's possible craft a completely legitimately looking image composed entirely of cross-origin resources; any pixel that the user is enticed to select may reveal cross-origin information in ways

If you can get only a color, not where it's from, how would the color reveal any information by itself in a otherwise legitimately looking way?

If you can get only a color, not where it's from, how would the color reveal any information by itself in a otherwise legitimately looking way?

@letitz described this scenario in the OP: in the simplest case, you tile the entire viewport with images/iframes pointing to a single cross-origin pixel.

That's like leaking up to a single pixel information on each user activation with a weird UI, and you still don't even know whether the color is from the content you control or something outside. How can that practically be a useful attack?

If this is a concern, then I suspect you should be more concerned about leaking user history via :visited style, because it would be easier to encode more information into a single color and guide user to click on it, and it would probably not be mitigated by cross-origin isolation.

Leaking of history via :visited is indeed a concern, and the eyedropper API is one way among many to leak such color information.

Fortunately, we have the ability to worry about multiple things at the same time.

I think for this issue, there are two questions:

1. How practical/serious is this attack vs. alternatives (e.g. forcing the user to reveal cross origin color data in some other way, e.g. social engineering)?

2. Are there mitigations that can be enabled to reduce the threat without significantly impairing the user-experience?

:visited indeed still very much needs to be solved, especially with Spectre. I think there have been some suggestions about scoping it to the top-level site which would largely address the issue.