WPF Key Forwarding from WebView2 to Host application?

[RickStrahl]

Is your feature request related to a problem? Please describe.

I'm looking to use the Web Browser control as part of an editor application that uses ACE Editor (Markdown Monster). Currently the application uses the Web Browser control and while it works I'm starting to run into issues with not being able to update libraries due to IE 11 support.

I've spent some time exploring WebView2 and for the most part it looks like I can duplicate the complex interop functionality with it. However, one sticking point is key forwarding:

The Web Browser control allowed me to pass through keyboard handlers for menu, toolbar and other keyboard maps from within the editor (ie. a focused element in the browser) to WPF. Meaning while in the editor I can press say (alt-w-l) to close all documents which is a WPF top level menu option. There's no extra code to make this works in the old control as the keystrokes are forwarded. This worked fine in the Web Browser, but has doesn't in WebView2.

AFAIK the old IE Web Browser control handles all locally mapped DOM key events and if those are not handled inside of the browser, the keystrokes are passed up into the WPF host which then passes on to the appropriate WPF keyhandlers.

Without this feature, keymapping is going to be really difficult as essentially every possible key combination has to be monitored and handled both in the browser and in WPF and is made more complicated yet by custom keymappings and the fact that this application allows for addins to add UI controls of their own that the core application may not know about (ie. adding a custom menu choice with it's own menu mnemonic).

Is there any chance that key forwarding can be added to the WebView2 to behave in a similar fashion to the IE control?

Describe the solution you'd like and alternatives you've considered

I would like this to work the same as it does in the IE Web Browser control:

• Any client side handled keyboard events are processed in the browser
• Any key events that go unhandled are passed up to WPF

Demonstrate how it doesn't work

• Create a WPF form.
• Add WebView2 control
• Add a navigation textbox
• Add a button and set <AccessText>_Navigate</AccessText>
• Add handler to set browser.Source = new Uri(txtUrl.Text)
• Navigate to a form that takes focus in the browser
• Type in some other URL in the form
• Focus the browser in the input control
• Press Alt-N

nothing happens.

AB#29558592

[champnic]

Thanks for checking #112. I'm tracking this on our backlog separately. Anything we build for our core Win32 control will also be projected in some way for our .NET controls, so it's possible we'll come up with a solution that solves both of these issues at once, but for now we'll keep both issues. Thanks!

[RickStrahl]

Any word on this?

I've been looking to see if there's some way to work around this, and I have a partial solution of keyforwarding from the DOM but it's pretty unreliable. Having unhandled key forwarding 'just work' as it did with the Web Browser control would be a much better solution.

I can't imagine I'm the only one that needs to be able to build interactive solutions that involve user input inside of the WebView and that need to still be able to interact with the host form UI.

[champnic]

If your main use-case are accelerator keys, then we've exposed the ICoreWebView2::add_AcceleratorKeyPressed in the regular WPF KeyDown event of the WPF WebView2 control. It doesn't capture every keypress though.

The ask for forwarding all key events is not near the top of our backlog, so will probably be sometime later in 2021 before we implement it.

[RickStrahl]

For me specifically it's the accellerators (Ctrl/Alt/Win/Func combos) that are important. It's possible that the old control only forwarded special key combos - never checked since the only thing I cared about was the menu/command forwarding ones.

I assume in WPF it's this:

<wv2:WebView2 AccessKeyManager.AccessKeyPressed="webView_AccessKeyPressed" />

 private void webView_AccessKeyPressed(object sender, AccessKeyPressedEventArgs e)
        {
            Debug.WriteLine(e.Key.ToString());
        }

But this doesn't seem to actually ever fire. Whatever key combos I fire don't show up in the debug window (while in an edit box - or in this case in ACE editor).

[ukandrewc]

@RickStrahl Until this is resolved you can hook the keyboard directly using 'SetWindowsHookEx', etc.

[RickStrahl]

The issue isn't so much capturing the keyboard commands. There are ways to look at all keys. The problem is getting the 'right' keys forwarded to WPF to process. Figuring out what keys were already handled, and putting together 3 key accelerator pairs is not easy to do at the app level. I started down this path with some experimental code and I'm getting lots of edge cases and false positives to pass on due to multi-event tracking and timing you have to do to even think about capturing all the keys needed.

This is why the old WebBrowser control was so nice both in WPF and WinForms: It handled all that for you - only passed through what wasn't handled, and passed all those keys onto WPF in a natural way. It just worked.

For Interop between the browser and a top level host application I think this is a really important scenario and one that I would prefer not to have to implement (probably incorrectly) in every end-user application.

Not saying this because I don't want to get my hands dirty, but because I think getting this right is really difficult. It's a system level task that requires deep knowledge about how keys are handled and injected into the key buffers. IOW, manual implementation is very likely to get this wrong almost every time, leading to badly behaving Windows applications. I think nobody wants that including Microsoft... :smile:

[champnic]

Sorry if my comment above wasn't clear. We are raising the corresponding KeyDown events for accelerators, so you should be able to do the following:

        private void webView_KeyDown(object sender, KeyEventArgs e)
        {
            if (e.Key == Key.P && (Keyboard.Modifiers & ModifierKeys.Control) > 0 && (Keyboard.Modifiers & ModifierKeys.Alt) > 0)
            {
                // Do something for Ctrl + Alt + P

                // Mark handled to cancel WebView2 handling
                e.Handled = true;
            }
        }

Let me know if that doesn't work for you, or you still have concerns about this approach. I've noticed a place in our code where we are missing hooking into the InputManager, which is potentially why AccessKeyManager isn't working here. I've opened a bug to follow-up on our side. Thanks!

[RickStrahl]

@champnic The problem is that it's not that easy as capturing a single key event. First the modifier combinations don't always come as both Alt-P down (for example) - some of it is Alt then W(indow) cLose for example. This doesn't capture in just the KeyDown event. You now have to capture key down track the alt key, then check for a period of time for other keys etc.

I say this because I started down this path to try to work around this - in my version I'm doing it inside of the DOM though to check for the modifier keys there (to avoid the JS -> .NET transition until needed) and only when state changes passing the keys to .NET. It works - sort of but even so it's still unreliable with some key combos.

What I'm getting at is that this is really not easy to implement at the app level. It's doable but to do it right takes a lot of effort. I'm suggesting that if it's possible to do this natively inside of the control and pass this forward in the same way to old controls did, that would avoid a lot of pain for a lot of developers. Any application that works with input inside of the Web Browser is likely going to need this functionality to truly integrate into a desktop application.

The other issue I see here with what you suggest is that intercepting every keystroke inside of .NET is likely going to cause a bit of overhead. I'm using a sophisticated editor (ACE Editor) in JavaScript and typing speed is rather important - I suspect sending every key event into .NET and checking for special keys is going to introduce some typing latency. Handled at the control (Win32 level) this is likely to cause much less overhead than the marshalled calls into .NET I suspect.

I think I've made my point here (I'm starting to repeat myself) - I get it if you don't want to implement this but I think there's a lot of value for a lot of developers in doing so even if it's kind of a hidden feature. When you need it, self-implementing is going to waste days of dev and testing to get it right.

[champnic]

Thanks for the added context @RickStrahl. Sounds like the base case of keyboard shortcuts/accelerators are working, but we're missing support for the mnemonics for menus (alt and then w, for example). I'll update our backlog scenario to reflect that and make it more clear. Does that capture the core of your ask?

[RickStrahl]

@champnic To be clear - I'm asking for key forwarding behavior similar to the old Web Browser control which forwards everything that isn't handled inside of the browser DOM to the shell. Take a look at how the old WB control works - there it 'just works'. Control shortcuts, menus etc. The only tricky ones are the multi-key mnemonics and key chords, but ultimately I'd like to see those just forwarded to the host if not handled in the DOM.

[mediabuff]

Webview2 needs to implement IKeyboardInputSink

https://docs.microsoft.com/en-us/dotnet/api/system.windows.interop.ikeyboardinputsink to interoperate with wpf

https://docs.microsoft.com/en-us/dotnet/desktop/wpf/advanced/windows-forms-and-wpf-interoperability-input-architecture?view=netframeworkdesktop-4.8

[RickStrahl]

@mediabuff Thanks for pointing out the specific APIs that handle this.

I really hope support for this makes it into the WebView2 control. The keyforwarding for me is a show stopper and prevents me from moving forward in the short term.

[mediabuff]

Absolutely! For many really world apps.

Also deep navigation integration - jointly with the hosting app on journal history - aka WFP/frame navigation is a must as well

[albahari]

Until 1.0.781, there was a nice workaround: you could handle the AcceleratorKeyPressed event, which exposes the WPARAM/LPARAM message properties, and then manufacture a windows message and send it to the main menu. Now that AcceleratorKeyPressed is no longer exposed, you must handle KeyDown, which doesn't expose the LPARAM property. If you know how to reconstruct the LPARAM from a KeyEventArgs, let me know.

[RickStrahl]

Ok, so here's an update on the key forwarding situation after a bunch more experimentation I think I found a solution to Menu and Toolbar shortcut forwarding issues.

After a bunch more experimentation I found that most 'special' keys are in fact forwarded into the host WPF form. Control keys and function keys seem to actually fire.

However alt keys that activate menus and shortcuts do not work and I think the reason for that is that they are getting sent but are getting ignored by the host Window because the window doesn't have focus. Because the window doesn't have focus the Alt key operation doesn't trigger the default Windows menu and shortcut activation functionality.

So it's possible to intercept the alt key via KeyDown on the WebBrowser control and then explicitly fire it into the WPF window after setting focus to it:

private void WebBrowser_KeyDown(object sender, System.Windows.Input.KeyEventArgs e)
{
    // Handle Alt-Key forward to form so menus work
    if (e.Key == System.Windows.Input.Key.LeftAlt)
    {
		Model.Window.Focus();
		
		// must be out of band
                Model.Window.InvokeAsync( () => SendKeys.SendWait("%"));

                e.Handled = true;
    }
}

Wrote up a blog post here:

WebView2: Forwarding Alt Keys to host WPF Window

I think this resolves the issue, but it still would be nice if this could happen automatically as it did for the old IE WebBrowser control so the extra key monitoring (which adds overhead especially for edit operations) isn't necessary.

[noseratio]

This keyboard issue is hindered by the fact that the WebView2 host's child window (created by HwndHost) and the actual hosted Chromium HTML renderer child window are residing in separate processes.

So, when the focus is inside a WebView2, the hosting WPF UI thread itself is focus-inactive, and the keyboard messages are posted to Chromium's own UI thread (in a separate process). Thus, the WPF menu accelerators and even some system hotkeys (e.g., Alt+Space for system menu) just don't work. The corresponding WM_KEYDOWN/WM_KEYUP/WM_SYSKEYDOWN/WM_SYSKEYUP messages simply don't reach the WPF thread's message queue.

As a workaround, I've played a bit with WH_KEYBOARD_LL, albeit without much success so far.

I'm able to reliably intercept all the desired keystrokes from the Chromium's thread (e.g., Alt+F for Menu/File). However, I can't automatically route them through the WPF hierarchy, without first setting the keyboard focus to the main window, i.e., Keyboard.Focus(Application.Current.MainWindow). This is similar to what @RickStrahl is doing above.

Some things I've tried, to get the keyboard events routed, while keeping the focus within the WebView2:

• Calling ComponentDispatcher.RaiseThreadMessage();
• Rising routed keyboard events with Application.Current.MainWindow.RaiseEvent();
• Calling IKeyboardInputSink.TranslateAccelerator().

As for IKeyboardInputSink.TranslateAccelerator, I believe, it's for reverse scenarios. I.e., when the container gives a hosted control a chance to handle a keystroke. That's conceptually close to IOleInPlaceActiveObject::TranslateAccelerator from the golden age of COM.

While the thing I'm looking for would be conceptually close to COM's IDocHostUIHandler::TranslateAccelerator (or IOleInPlaceFrame::TranslateAccelerator, or IOleControlSite::TranslateAccelerator). Basically, a control asking its container to process a specific keystroke.

Logically, IKeyboardInputSite would be the right place for this, but it doesn't have anything like TranslateAccelerator.

IDK, maybe WPF has a proper secret solution for this scenario, but I couldn't find one. I wish WebView2 was open-source, it might be easier to tackle this problem from that end.

Anyhow, I'd be reluctant to convert a WPF project to use WebView2 at this stage, until we could reliable use accelerator keys on native WPF controls. We shouldn't have to go through some sort of mumbo-jumbo workarounds for such a basic thing like using keyboard for UI navigation.

[szanto90balazs]

@champnic I read about all the keyboard event related issues and wondering are they on the roadmap to tackle them for the near future and do you have a sneak peak into what changes likely / unlikely coming that you can share with us? Many thanks 👍 I'm particularly interested in the scenarios, when the WebView2 is the main input sink, but any unhandled DOM key events are surfaced to the WPF world. Good example would be to let the HTML/JS document add event handlers for Ctlr+SomeChar override WPF events (Ctrl+Somechar input binding).

[champnic]

@szanto90balazs We are currently in the design phase of this work, but have a good direction forwards here. We are going to change the main input sink from the WebView2 out-of-proc HWND to be the in-proc app HWND, which should enable full support for keyboarding, mouse, etc. in WPF and Winforms :) It will require some additional changes to our controls (to make sure KeyDown isn't getting fired multiple times), but should fix a ton of issues and enable new capabilities.

[szanto90balazs]

@champnic Sounds wonderful, thanks for sharing, much appreciated.

[SequentialCode]

Ok, I am late to the party but I have the same requirement for WebView2 embedded in my MFC application. I have the handle to the parent window and I want parent window to take care of keyboard shortcuts. Please consider this @champnic . Also, from the example in https://docs.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2controller?view=webview2-1.0.1150.38 for add_AcceleratorKeyPressed, what is m_appWindow->GetAcceleratorKeyFunction() referring to? Please give a similar workaround for C++ temporarily to handle keyboard shortcuts such as Ctrl + N, Ctrl + etc.

[champnic]

@hakzhyena The m_appWindow sample code is taken from our sample app: https://github.com/MicrosoftEdge/WebView2Samples/tree/master/SampleApps/WebView2APISample

You can use the add_AcceleratorKeyPressed event to handle basic keyboard shortcuts for now.

[SequentialCode]

Hi @champnic , thanks for the input. I was able to get the Ctrl key combinations working but as @RickStrahl has mentioned above, the Alt keys work little differently. I followed his approach and following is my code in C++:

if (command == "IDM_ALT_PRESS")//at this point Alt key press is already captured and args->put_Handled(TRUE) is also done so that browser doesn't handle it again 
{
MainWindow()->SetFocus();
MainWindow()->PostMessage(WM_KEYDOWN, VK_MENU);
return;
}

Interestingly it works when I press Alt two times when WebView2 is in focus and then it toggles but for the first time it doesn't work. Any ideas?

[SequentialCode]

Hi @champnic , any thoughts on the issue above please?

[champnic]

Unfortunately no. You could open a separate GitHub issue to track that if you like. We are currently working on a change which would make input go to the host windows first before the WebView2 which would make this scenario much easier, but I don'thave a clear timeline on when that will ship yet.

[SequentialCode]

@champnic Thanks for the response, I have created a separate GitHub issue for tracking.

[FStockinger]

Hello,

thanks for this solid project!

On behalf of my employer I'd like to mention that I wish for a proper solution for this issue as well. What I need is basically a mechanic "Forward all key presses to the host application and let it decide whether it handles the key, or whether the browser can do with it what it wants." Proper event handlers are needed, that can be added to the ICoreWebView2/Controller/Environment . All of this for WebView2 embedded in an MFC application. I now use a workaround, that I consider dirty.

Cheers

[leaanthony]

@szanto90balazs We are currently in the design phase of this work, but have a good direction forwards here. We are going to change the main input sink from the WebView2 out-of-proc HWND to be the in-proc app HWND, which should enable full support for keyboarding, mouse, etc. in WPF and Winforms :) It will require some additional changes to our controls (to make sure KeyDown isn't getting fired multiple times), but should fix a ton of issues and enable new capabilities.

Hi @champnic - just wondering where we are with this. Many thanks 🙏

[champnic]

Hey @leaanthony - It looks like the majority of the code has been implemented and is going through a code review now. This is relatively complex and large change, so may take a while before it gets checked in (ie. up to a couple weeks, not a day). I think the intent is also to have it be an opt-in behavior change, to avoid app-compat issues. Thanks!

[pareekvipul]

Hi @champnic - We are also facing the same issue with the key down events so can you please provide us the ETA for your fix release or prerelease.Thanks!

[champnic]

It's still in code review - I'm reaching out to the owners for updated status. Thanks!