WindowsAppSDK
WindowsAppSDK copied to clipboard
microsoft
On the Windows app SDK using CameraHelper, FrameArrived time execution for a period of time will not be triggered.
Describe the bug
When I use wasdk 1.1.2, the processing of camera frame data, at first the FrameArrived event is normal, but after a period of time, the event does not trigger, I suspect it is was wasdk bug, but I will confirm it here first. The uwp code is normal like this https://github.com/GreenShadeZhang/GreenShade.UWPDemo/tree/master/GreenShade.ML.EmoticonDetection
But when the code is ported to winui, it can only be normal for a while.
https://github.com/GreenShadeZhang/GreenShade.UWPDemo/tree/master/GreenShade.WinUI.EmoticonDetection
Citation issues https://github.com/CommunityToolkit/WindowsCommunityToolkit/issues/4606
So I wonder if using camera operations on wasdk is not yet functional, or when will it be possible to support completeness?
Steps to reproduce the bug
- Download the code
- Run the relevant code
Expected behavior
No response
Screenshots
No response
NuGet package version
1.1.3
Packaging type
Packaged (MSIX)
Windows version
Windows 11 version 21H2 (22000)
IDE
Visual Studio 2022
Additional context
No response
This issue still happens, even on WASDK 1.2.230118.102. The firing of the FrameArrived event of the CameraPreview toolkit control is inconsistent and will eventually stop.
private void CameraHelper_FrameArrived(object sender, FrameEventArgs e) { SoftwareBitmap MySoftwareBitmap = e.VideoFrame.SoftwareBitmap; SoftwareBitmap TargetSoftwareBitmap = MySoftwareBitmap; if (MySoftwareBitmap != null) { if (MySoftwareBitmap.BitmapPixelFormat != BitmapPixelFormat.Bgra8 || MySoftwareBitmap.BitmapAlphaMode == BitmapAlphaMode.Straight) { TargetSoftwareBitmap = SoftwareBitmap.Convert(MySoftwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied); if (TargetSoftwareBitmap != null) { CapturedImage.DispatcherQueue.TryEnqueue(DispatcherQueuePriority.High, async () => { SoftwareBitmapSource SBS = new(); await SBS.SetBitmapAsync(TargetSoftwareBitmap); CapturedImage.Source = SBS; // Image XAML control }); } } }
MediaPlayerElement should be fixed.
Seems like this is still an issue in 1.3.x as well: https://github.com/CommunityToolkit/Windows/issues/108#issuecomment-1624349364
Otherwise, our Toolkit helpers would seem to work fine, but are essentially useless due to this underlying platform issue.
I found some precautions in the uwp demo given in the official document. It is mentioned in the document that if the data is not released in time, the frame arrival event will not be triggered normally, and I used the official uwp demo to transform it into winui At that time, it was indeed normal, but I don't know what operations the official demo did.
I suggest that you can refer to the demo I used, and then see if you can process the code for the corresponding platform separately.
https://github.com/GreenShadeZhang/WinUI-Tutorial-Code/tree/master/src/CameraFrames
@jonwis @bpulliam any updates on this issue, will this make being fixed in the upcoming 1.4?
This is still broken in the 1.4 stable release.
I don’t know if it’s because of the framework, or if the code requires special processing.
This is still broken in the 1.4 stable release.
I don’t know if it’s because of the framework, or if the code requires special processing.
The framework is the issue here.
Related: microsoft/microsoft-ui-xaml#4710
I would like to know whether this issue is a bug in the windows app sdk or a bug in the xaml ui. At present, this issue seems to have been put on hold. I really hope that this function can be used normally, and I hope that the official will give the latest plan.
It is still not working, even when I use WAPSDK 1.5.* and the latest 970 build of camera preview.
Current behaviour: Webcam output is shown, updates for a (few?) frames and output freezes.
It is still not working, even when I use WAPSDK 1.5.* and the latest 970 build of camera preview.
Current behaviour: Webcam output is shown, updates for a (few?) frames and output freezes.
I think the team thinks it is a problem with another team's library, so now it seems that internal cross-department coordination is needed to confirm where the problem is. This function is very important and I hope it can be solved.
I believe I have found the source of this issue. As explained on the MediaFrameReader docs,
If you access the SoftwareBitmap or Direct3DSurface objects provided by the VideoMediaFrame property of a MediaFrameReference, the system creates a strong reference to these objects, which means that they will not be disposed when you call Dispose on the containing MediaFrameReference. This also applies to the SoftwareBitmap or Direct3DSurface objects provided by the VideoFrame object that you obtain from the MediaFrameReference.VideoMediaFrame property. You must explicitly call the Dispose method of the SoftwareBitmap or Direct3DSurface directly for the objects to be immediately disposed. Otherwise, the garbage collector will eventually free the memory for these objects, but you can't know when this will occur, and if the number of allocated bitmaps or surfaces exceeds the maximum amount allowed by the system, the flow of new frames will stop.
The Windows Community Toolkit CameraHelper mentioned in this issue calls GetVideoFrame. If the user does not call dispose on the VideoFrame, it will stop working as described in the passage above. Unfortunately, the CameraHelper calls GetVideoFrame even if you are using the CameraPreview control, which does require the use of the FrameArrived event and will leak and stop working if you do not dispose the frame yourself.
My conclusion is that there is no issue with Windows App SDK, XAML, or Windows, and Windows Community Toolkit is just not disposing the frame that it is loading.
I believe I have found the source of this issue. As explained on the MediaFrameReader docs,
If you access the SoftwareBitmap or Direct3DSurface objects provided by the VideoMediaFrame property of a MediaFrameReference, the system creates a strong reference to these objects, which means that they will not be disposed when you call Dispose on the containing MediaFrameReference. This also applies to the SoftwareBitmap or Direct3DSurface objects provided by the VideoFrame object that you obtain from the MediaFrameReference.VideoMediaFrame property. You must explicitly call the Dispose method of the SoftwareBitmap or Direct3DSurface directly for the objects to be immediately disposed. Otherwise, the garbage collector will eventually free the memory for these objects, but you can't know when this will occur, and if the number of allocated bitmaps or surfaces exceeds the maximum amount allowed by the system, the flow of new frames will stop.
The Windows Community Toolkit CameraHelper mentioned in this issue calls
GetVideoFrame. If the user does not call dispose on the VideoFrame, it will stop working as described in the passage above. Unfortunately, the CameraHelper calls GetVideoFrame even if you are using theCameraPreviewcontrol, which does require the use of theFrameArrivedevent and will leak and stop working if you do not dispose the frame yourself.My conclusion is that there is no issue with Windows App SDK, XAML, or Windows, and Windows Community Toolkit is just not disposing the frame that it is loading.
@michael-hawker Can the solution mentioned above be baked-in to the toolkit? Thanks.
@michael-hawker Can the solution mentioned above be baked-in to the toolkit? Thanks.
Tracking in https://github.com/CommunityToolkit/Windows/issues/375#issuecomment-2018699511