Picture-in-Picture for any Element

[mounirlamouri]

The spec was designed to be extensible to any element (not only HTMLVideoElement). @beaufortfrancois and I had a look today and it seems that at the API level, the main change would be to have the partial interface HTMLVideoElement be a partial interface Element. disablepictureinpicture makes a bit less sense for Element but could be kept for consistency. We would need some special steps for videos like having metadata and a video track.

Edit: to clarify something, there would be not that many change in functionality and this is something we are still looking into all of this but we believe it could avoid having to create a to do post processing for cases where a website wants to show some UI on the Picture-in-Picture. The idea is still to have the window non interactive so it should not affect security.

We are still looking into what would be required in order to easily allow websites to send random parts of their DOM to Picture-in-Picture but thought we should open a bug in case of @jernoble, @jyavenard or others have an opinion.

[scottlow]

@mounirlamouri @beaufortfrancois do you mind sharing a brief summary of the security concerns that have been discussed already with respect to interactive HTML content? With this proposal on the table, I was thinking it may be worth revisiting #69 in parallel, but wanted to make sure I had a full understanding of the existing concerns first.

[mounirlamouri]

Picture-in-Picture is a popup window that is always on top and has no chrome which could allow a website to pretend to be something else by creating a popup window. We can imagine this being abused to scam users as a website pretend to be something else (like a security warning offering an AV software).

This could probably be resolved somehow but there is another issue which is that arbitrary interactive Picture-in-Picture wouldn't be compatible with iOS and Android. We may not want to design an API that wouldn't work on most mobile systems.

[scottlow]

Gotcha, thanks. I figured phishing-related attacks were the main concern, but figured I'd ask in case there were other scenarios that had been identified.

I agree that we should be cognizant of both desktop and mobile support in our design, however, based on developer interest in the scenarios that fully interactive PiP could enable, it seems like something worth considering.

What are your thoughts on the addition of a read-only document.pictureInPictureInteractable boolean that could be set by the UA to indicate whether it supports fully interactive PiP? This could then be used as follows to determine whether an interactive or non-interactive PiP experience should be displayed:

<div id="interactivePiPContent"></div>
<div id="nonInteractivePiPContent"></div>

<button id="togglePipButton"></button>

togglePipButton.addEventListener('click', function() {
    if (document.pictureInPictureInteractable) {
        interactivePiPContent.requestPictureInPicture();
    } else {
        nonInteractivePiPContent.requestPictureInPicture();
    }
}

Also, I can open a new issue for full interactivity if you'd like to move this conversation there and keep this one focused on allowing picture in picture on Element.

[mounirlamouri]

Thanks for your feedback. I think it's the right approach to have a way to feature detect whether interactive PIP is supported by the platform. Something I wonder from your proposal is how doable it is to display a

in a different window while being able to easily propagate the UI events. It seems to me that if we want to go that way, it may be more realistic to have a window.open()-like API if not simply extending window.open(). It may make sense to continue this discussion in a different issue. We should also use the face to face after FOMS to talk about it more.

[scottlow]

@beccahughes from the v2 explainer, it's not immediately clear to me how a site requesting interactive PiP content is meant to "easily fall back to non-interactive content" in the event that the platform does not support it.

Is the idea that the promise returned by calling requestPictureInPicture on an HTMLElement will reject if interactive is set to true on a platform that does not support interactive PiP content?

[beccahughes]

Yes this is the case. With promise chaining something like the following should work:

interactiveUI.requestPictureInPicture({ interactive: true })
  .then((w) => return w)
  .catch(e => return nonInteractiveUI.requestPictureInPicture({ interactive: false }))
  .then((w) => {
    w.document.body.appendChild(videoElement);
  });

[ewoerner]

The spec was designed to be extensible to any element (not only HTMLVideoElement). @beaufortfrancois and I had a look today and it seems that at the API level, the main change would be to have the partial interface HTMLVideoElement be a partial interface Element.

I believe there should be another API change: Since most Elements cannot provide any "natural" aspect ratio on their own, it should be possible to pass the suggested aspect ratio along using requestPictureInPicture({ aspectRatio: 4/3 }) or something similar.

[derN3rd]

@ewoerner you mentioned aspect ratio, do you (or anyone else here) aware of a solution to have the PiP window fixed for a ratio, so resizing doesn't show black borders left/right or top/bottom?

Already tried looking into the onresize and resizing the video, but that didn't work out. Also pip props are readonly

[aarongustafson]

On Twitter today, @gregwhitworth drew a connection between this and the "projections" idea for Progressive Web Apps I’ve been noodling on for a while. I just wanted to flag that up in case folks have thoughts about where they might overlap.

WICG Discussion: https://discourse.wicg.io/t/noodling-on-an-idea-projections-for-web-apps/3900/

[ClassicOldSong]

I think it's just like the requestFullScreen API, almost all of the things are the same including interaction security issue. requestFullScreen forces your elememt showing as your monitor's size and aspect ratio, so forcing the pip element showing as user dragged size should be fine.

[mubashir-rehman]

Hi everyone. Hope you are doing well. I was looking for a solution to pop out a chat from meet app by google. So that i can see the chat even when I am presenting other tabs. We use google meet for our classes. It is hard to interact with our teacher because they can not see our chat when they are using other apps. It becomes annoying when they see our message after 2 min of explanation about we didn't understand the previous concept. Any suggestions are accepted even if you reply that "this is irrelevant to the topic" would be great.

[keyfer]

Reaching out to say that doing this would be incredibly useful. I was looking into requestPictureInPicture for the "video" frame of a Google Meet, and because of how the Meet code structures their participant video data, it's impossible.

This would be really useful for being able to PiP both users with active videos, as well as users with video off, in the same PiP.

[1valdis]

Hey there. My usecase is I want to enable users to take actions in certain moments of time in another app, by hand. So I want to show a floating picture-in-picture which can contain precise current time. This could certainly be a useful feature.

[namgk]

Hi there,

I've just opened another issue about Canvas support: https://github.com/w3c/picture-in-picture/issues/180.

I think canvas is a special element cause we can capture media tracks from it, effectively like a video element.

My concern on that issue is if the canvas is animated via requestAnimationFrame, it is paused when the tab is inactive. I'm not sure if the pop up causes the tab to be not in inactive mode?

[Ademking]

@keyfer i found a google extension called "Google Meet Pip" https://chrome.google.com/webstore/detail/google-meet-pip/ibkkdnfblcekolaagnljieekcaccfpjo

[fluidsonic]

Despite the title this issue seems to revolve around discussing an interactive version of PiP.

Couldn't this be split up into two issues/changes? Supporting PiP as-is for any element seems to be doable rather quickly while an interactive solution needs much more thought. Those features could be added one after the other which would avoid delaying the non-interactive solution.

[mkhahani]

This feature could be very useful for many use cases. Any plan?

[chrisn]

@mkhahani Please take a look at the proposed Document Picture-in-Picture API, see https://discourse.wicg.io/t/proposal-document-picture-in-picture/5736 for details and consider mentioning your use cases there.

[beaufortfrancois]

Closing all v2 labels issues as this is replaced by the dedicated WICG Document Picture-in-Picture API proposal.