libobs-winrt: Check device state in on_frame_arrived

[walker-WSH]

Description

if D3D11Device is not in normal status, frame_pool.Recreate will crash. frame_pool.Recreate will be called frequently during resizing window dialog with WGC method.

From PRISM's cloud logs, we see that many users crashed at this location after device_removed/device_reset (no rebuild yet!). We can also to reproduce this crash if we modify the code to ensure Recreate is called before rebuilding

Motivation and Context

avoid crash caused by frame_pool.Recreate

How Has This Been Tested?

1. add window source with WGC capture method.
2. use "dxcap -forcetdr" to trigger device error during resizing window

Types of changes

bug fix (non-breaking change which fixes an issue)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [x] I have included updates to all appropriate documentation.

[Lain-B]

Does winrt_capture::on_closed() just not get called before that happens? I don't mind having a check for device invalidation, but I think we should shut down the source entirely if that happens.

[walker-WSH]

@Lain-B

on_closed and on_frame_arrived (where frame_pool.Recreate is called) are both called from graphic thread. If exception of dxdevice occurres during calling on_frame_arrived, and Recreate is called at that time, then the crash happen.

---

We know the logic flow is as below in graphics thread:

1. tick_sources
2. TranslateMessage -> on_frame_arrived will be called
3. render_displays -> rebuild may be triggered by swap->Present or swap->ResizeBuffers -> notify rebuild lib-winrt

Generally, the "shut down" you said of lib-winrt is done in step-3 by below callbacks (called from gs_device::RebuildDevice):

However, sometimes in step-3 there is no dxdevice error, and it happened in next step-2 and Recreate is about to be called. Only in this scenario， crash can be reproduced.

The important reason:

If dxdevice is in wrong state, frame_pool.Recreate will crash.

That is to say, dxdevice has been wrong state, but before rebuilding in step-3, frame_pool.Recreate is called in step-2, then crash happened! We can modify code to trigger this scenario which make crash reproduced.