Picture in Picture support exploration

[tymscar]

One thing that I personally would love to do would be to have the option of leaving my stream picture in picture while I do something else on my phone. Most of the time when I use Moonlight I actually don't play games, but use it as a portable screen for my computer. It's great in meetings, or when I want to quickly check on long-running programs/processes. I've seen others mention PiP before, in threads such as these ones: https://github.com/moonlight-stream/moonlight-ios/issues/84 https://github.com/moonlight-stream/moonlight-ios/issues/496

The best solution really would be to be able to swipe home, and then PiP start in a corner as it does in other apps. It would disable interaction obviously, but then you could always go back to the full-screen experience when you need to interact.

As for alternatives, I'm not sure there's much really that could work as one. PiP is an officially supported feature on iOS.

So with this in mind, I had some free time this weekend, so I tried to implement it myself, but I have failed. I'm sure there are much smarter people here than me and maybe if we band together we can figure it out!

What I've managed to accomplish was successfully enabling the actual PiP protocol in the app:

• Modified Limelight-Info.plist: Set UIRequiresFullScreen to false and added audio to UIBackgroundModes
• Inside VideoDecoderRenderer, I've implemented an AVPictureInPictureController
• Added PiP delegate methods to handle playback state
• Modified StreamFrameViewController to start PiP instead of terminating when backgrounded (part of what was also asked by others to keep the stream going when backgrounded).
• PiP window successfully appears and displays the stream briefly (but we are talking a handful of frames if that)

I think that disconnection is due to the control stream timeout. I'm no expert in ENet-based control streams, but this is a super low latency thing (obviously needed for gaming), and the way iOS backgrounds apps, it basically suspends network threads. Then the control stream can't send the keep-alive packets, and then after like a second or two, the server disconnects, at which point the connection terminates.

I know there's obviously tons of apps like YouTube and Twitch that support PiP, but I think their streaming setup is much less custom, and it just uses HTTP/HLS streaming that's probably handled by iOS natively in the background? (just a guess). I reckon they also buffer many, many more frames in advance.

So overall, I think enabling the PiP ui, handling the callbacks, and starting it when backgrounded are super easy to do. I just wonder if we can figure out a way to keep the ENet control stream alive in the background, and maybe look into how iOS keeps things running now with these new versions. If I remember correctly, there was talk about supporting more intense background tasks lately.