WebView2Feedback Off-screen rendering with webview2

Hi Is it possible to get webview2 to render into a shared memory region like CEF : https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-header-off-screen-rendering

In my application I use 2 process architecture where main process does not have network access and second process uses CEF to render webcontent into a shared memory region from where main app can read the pixels. I am wondering if I can achieve this using webview2

Thanks

AB#28491736

Oct 20 '20 06:10 ajaymonga

Currently this is not possible. We have offscreen rendering on our backlog, and are tracking it in #20, but I think this ask is clearer so I'll also add this issue to our item. Thanks!

Oct 20 '20 20:10 champnic

Hi, do you have any updates on this which you're able to share?

Oct 10 '21 16:10 Harvey3141

Unfortunately not yet. We have not begun work on this yet. This is a large amount of work, and while very high on our priority list, gets bumped each quarter as higher priority asks come in. I really want to do this work as it's currently one of our top asks, but the earliest that could happen is Q1 2022 at this point.

Oct 11 '21 20:10 champnic

Any news on this?

Apr 08 '22 08:04 avail

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

Apr 08 '22 19:04 champnic

We are currently using CefSharp to render into an offscreen buffer that is sent to an FPGA to be composited onto a live video feed. The render includes playback of MPEG4-encoded video. We were looking into using WebView because CefSharp's default Chromium build does not include the MPEG4 codec. But without offscreen rendering, that's a moot point. We've since produced our own custom build of Chromium to include the codec.

Apr 08 '22 21:04 doxxx

Our use case is to show web content in our immersive VR spaces. See https://www.igloovision.com/software/enterprise-package/igloo-web

We need access to the web rendered textures so that we can warp and blend multiple images to projector outputs to create a clear seamless view on the inside of a cylinder or room.

Currently we use CEF for the web input with a user maintained patch for getting the shared textures. See https://bitbucket.org/chromiumembedded/cef/pull-requests/285 and read down to the end of the comments for full details. We build CEF with proprietary codecs included. The builds of our app and CEF are done with C++.

This is unsatisfactory because it uses the deprecated custom compositing (which could be removed in future) instead of skia and is difficult to keep updated.

We would look at moving to webview2 if access to the rendered textures could be provided and supported along with the ability to include proprietary codecs

Apr 13 '22 11:04 rjx-ray

Thanks for the info @rjx-ray! If you don't need high-frequency rendering you could consider using the CapturePreview function to get an image, but this isn't great for things like videos or other animations.

Apr 13 '22 17:04 champnic

Hi @champnic, thanks for the response but we do need high-frequency rendering, typically for YouTube and other web video display.

Apr 13 '22 19:04 rjx-ray

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

Game UI development, specifically within DirectX 9 (Ex) / 10 / 11 contexts.

Ability to render into an offscreen texture and display that as overlay of the game.

Apr 17 '22 01:04 avail

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

Currently, we utilize CefSharp Offscreen for generating a PDF from HTML content or using it to generate screenshots of HTML in a server side environment.

May 11 '22 15:05 jcain82

Hi, are there any updates on this feature?

Jul 17 '22 00:07 zziger

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

CAD editor software, I want to compose web ui and opengl/d3d rendering together.

Jul 21 '22 13:07 lencil

Any XR Application wanting to be able to have some 2D UI inside the 3D Environment: it means being able to render into a DirectX Texture and to be able to inject input in the off-screen view (pointer + keyboard)

Jul 21 '22 15:07 alainza

Thanks for the info! Unfortunately our design work is slow going due to high priority bugs, but we're still making progress.

Jul 21 '22 18:07 champnic

Is #579 related?

Jul 27 '22 08:07 danielvandenberg95

My use case would be Game Overlay. And actually dotnet/maui(and in extension WPF/WinForms)'s BlazorWebView could benefit from this change too as they wouldn't have to create the fake elements(they are just precisely tracking the WebView window on top), but could do normal real elements in their respective apis without any weird quirks and bugs.

Jul 27 '22 23:07 honzapatCZ

I'd say an api like CefSharp would be quite pleasant and easy to port over existing CefSharp code. https://github.com/cefsharp/CefSharp/blob/d8674fd076c021eddcc0cb579687ca3c51a63767/CefSharp.OffScreen/DefaultRenderHandler.cs

Jul 27 '22 23:07 honzapatCZ

Hi, do you have any updates you're able to share?

Nov 03 '22 09:11 Harvey3141

Adding another use case: Presentations on screens that are not just Windows desktops.

From our experience with CEF we'd need the following (in descending priority order) to switch:

Rendering into either a raw CPU buffer or into a fully accelerated DXGI surface (higher level APIs optional but this is what most people use)
Full control over the presentation parameters, such as resolution, frame rate, display scaling, output color space (including WCG/HDR), etc. Fully manual VSync is appreciated; the client engine might render ahead and buffer, or batch render to disk, so it's not strictly in real time.
Proper graphics resource lifetime management. Client app must be able to exactly specify when/how it is able to receive frames and when it is finished processing them; no spurious callbacks when client isn't ready (anymore), and no mutexes that render either side unresponsive when held a bit too long or not acknowledged (that one probably came out a bit too detailed, but you might be able to feel the pain here).
Full manual input injection, such as keyboard, mouse, and multi touch inputs.
Audio should also be redirected. Ideally per browser instance but globally would be fine at first. Client must be able to specify the sample format and channel layout, and it should be a pull model in which the client requests a certain # of sample frames and WebView2 outputs that number of frames exactly, in order to avoid timing discontinuities between the browser audio and the actual audio path the client uses.
... and you might recognize a pattern here. There are other subsystems that would be cool to have redirected to an API, for example audio/video input (for web based streaming/conference services) or geolocation. A feature to auto-block all outside interaction that's not intercepted (Midi, Bluetooth, notifications, printing (possible security hole there!), etc) would perhaps be helpful, too.

It might be tempting to keep these APIs as close to the corresponding Windows APIs as possible but it's not strictly necessary - this feature will mostly be used from deep within client code, and very possibly behind a bunch of abstraction layers, so I'd aim for minimal and clean first.

Hope this was not too much/harsh but currently a whole industry is dependent on that CEF pull request above which has been in PR limbo for years and will soon just stop working altogether, and an alternative to that would be a very appreciated thing ;)

Nov 04 '22 17:11 VentuzTammoHinrichs

@champnic - Is there any news you can share with us?

Nov 15 '22 13:11 DanielsCode

+1

Nov 15 '22 14:11 robotsinthesun

+1

Please use the thumbs up reaction instead of occluding the issue with +1 comments

Nov 15 '22 20:11 Curve

@DanielsCode Unfortunately not really. We had begun the design phase but are currently dealing with high priority bugs. I really want to get this done as I know it's a huge pain for a large portion of our developers, and I'm hoping we can get back to focusing on it soon.

Nov 17 '22 00:11 champnic

Hello, just to add another use case.
I would like to create a ASP Web API controller where the user POSTs an HTML string, or JSON object that is merged into an HTML template, the controller would use WebView2 to render the HTML off-screen, save a PDF to a stream and then return the stream as a file for the user to download.

It seems CoreWebView2.PrintToPdfStreamAsync is in pre-release, so we just need headless mode and I'm good to go.

Feb 01 '23 16:02 LukeTOBrien

Hello,

Using WinForms, has anyone tried to put the WebView2 component on a separate Form and put that form off-screen or make it not Visible?

With either of this code:

        private void OnFormLoad(object sender, EventArgs e)
        {
            Form form = (Form)sender;
            form.ShowInTaskbar = false;
            form.Visible = false;
        }

        private void OnFormLoad(object sender, EventArgs e)
        {
            Form form = (Form)sender;
            form.ShowInTaskbar = false;
            form.Location = new Point(-10000, -10000);
        }

Was able to use it this way but, since it's not an "official" way, anyone that has used it has detected any kind of issues with it?

The first bug I noticed is when doing Alt+TAB with the form on offscreen it shows up on the list, with the window not visible it flickers when it opens.

Thank you