Unity Mousewheel crash. How do I implement scroll pages in HoloLens 2 Unity apps?

[PurpleJoy]

I went through the tutorial and downloaded the examples.

Following the example, I implemented OnPointerDown() and OnPointerUp().

But found that can only achieve click operations and can not scroll the web page.

I want to make it easy for users to scroll and navigate the web using gestures. How do I do that?

(I tried to implement MouseWheel operation in OnPointerDragged(), but couldn't find any instructions or examples, and every time I tried to send a MouseEvent myself, Unity crashed.)

Here is part of my code :

bool isMouseDown = false;
Vector3 startPos;

public void OnPointerDown(MixedRealityPointerEventData eventData)
{
    TranslateToWebViewMouseEvent(eventData, WebViewMouseEventData.EventType.MouseDown);
    UnityEngine.Ray leftHandRay;
    bool isGetSuccess = InputRayUtils.TryGetHandRay(Microsoft.MixedReality.Toolkit.Utilities.Handedness.Right, out leftHandRay);
    if (isGetSuccess)
    {
        startPos = leftHandRay.origin;
    }
    isMouseDown = true;
}

public void OnPointerUp(MixedRealityPointerEventData eventData)
{
    TranslateToWebViewMouseEvent(eventData, WebViewMouseEventData.EventType.MouseUp);
    isMouseDown = false;
}

private void TranslateToWebViewMouseEvent(MixedRealityPointerEventData eventData, WebViewMouseEventData.EventType evenType)
{
    var hitCoord = NormalizeWorldPoint(eventData.Pointer.Result.Details.Point);

    hitCoord.x *= _webView.Width;
    hitCoord.y *= _webView.Height;

    var mouseEventsWebView = _webView as IWithMouseEvents;
    WebViewMouseEventData mouseEvent = new WebViewMouseEventData
    {
        X = (int)hitCoord.x,
        Y = (int)hitCoord.y,
        Device = WebViewMouseEventData.DeviceType.Pointer,
        Type = evenType,
        Button = WebViewMouseEventData.MouseButton.ButtonLeft,
        TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
    };

    mouseEventsWebView.MouseEvent(mouseEvent);
}

private Vector2 NormalizeWorldPoint(Vector3 worldPoint)
{
    // Convert the world point to our control's local space.
    Vector3 localPoint = transform.InverseTransformPoint(worldPoint);

    var boundsSize = Collider.sharedMesh.bounds.size;
    var boundsExtents = Collider.sharedMesh.bounds.max;

    // Adjust the point to be based on a 0,0 origin.
    var uvTouchPoint = new Vector2((localPoint.x + boundsExtents.x), -1.0f * (localPoint.y - boundsExtents.y));

    // Normalize the point so it can be mapped to the WebView's texture.
    var normalizedPoint = new Vector2(uvTouchPoint.x / boundsSize.x, uvTouchPoint.y / boundsSize.y);

    return normalizedPoint;
}

public void OnPointerDragged(MixedRealityPointerEventData eventData) 
{
    if (isMouseDown)
    {
        UnityEngine.Ray leftHandRay;
        bool isGetSuccess = InputRayUtils.TryGetHandRay(Microsoft.MixedReality.Toolkit.Utilities.Handedness.Right, out leftHandRay);
        if (isGetSuccess)
        {
            var distance = Vector3.Distance(startPos, leftHandRay.origin);
            if (distance > 0.01)
            {
                var mouseEventsWebView = _webView as IWithMouseEvents;
                WebViewMouseEventData mouseEvent = new WebViewMouseEventData
                {
                    WheelX = 0,
                    WheelY = distance,
                    Device = WebViewMouseEventData.DeviceType.Pointer,
                    Type = WebViewMouseEventData.EventType.MouseWheel,
                    Button = WebViewMouseEventData.MouseButton.ButtonMiddle,
                    TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
                };

                mouseEventsWebView.MouseEvent(mouseEvent);
            }
        }
    }
}

AB#46192317

[champnic]

@michaelfarnsworth could you please take a look?

[RubenGarcia]

I'm using Webview2 0.17.1.pre3 and I don't have Device in

        Device = WebViewMouseEventData.DeviceType.Pointer,

What version are you using? I don't see Device at https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/webview2-plugin-mouse-event-data-class

[michaelfarnsworth]

@RubenGarcia that's odd. There should be no published version without that enum defined. It's under the Microsoft.MixedReality.WebView namespace. Are you just trying to build the GettingStarted sample app?

[PurpleJoy]

@RubenGarcia I'm using version 0.17.1-Pre.5. This is the example I mentioned earlier: https://github.com/MicrosoftEdge/WebView2Samples/tree/main/GettingStartedGuides/HoloLens2_GettingStarted. In this example, WebViewBrowser.cs uses Device.

[RubenGarcia]

I had issues with Pre.5; I'll test it again and report.

[RubenGarcia]

I confirm that removing the Device line works on Pre.3, and clicks are correctly received and processed by the webpage.

[RubenGarcia]

I confirm that using your code to create MouseWheel events crashes Unity, both in OnPointerDragged and on Update(). However, using a MouseWheel event mapped to a keypress seems to work. I suspect that calling a MouseWheel event too often (before the previous one finishes executing) is causing the issue.

[PurpleJoy]

I confirm that using your code to create MouseWheel events crashes Unity, both in OnPointerDragged and on Update(). However, using a MouseWheel event mapped to a keypress seems to work. I suspect that calling a MouseWheel event too often (before the previous one finishes executing) is causing the issue.

I tried to create MouseWheel events in OnPointerDown(), Unity still crash. How did you make it work? Here is my code:

public void OnPointerDown(MixedRealityPointerEventData eventData)
{
    var mouseEventsWebView = _webView as IWithMouseEvents;
    WebViewMouseEventData mouseEvent = new WebViewMouseEventData
    {
        WheelX = 0,
        WheelY = -0.5f,
        Device = WebViewMouseEventData.DeviceType.Pointer,
        Type = WebViewMouseEventData.EventType.MouseWheel,
        Button = WebViewMouseEventData.MouseButton.ButtonMiddle,
        TertiaryAxisDeviceType = WebViewMouseEventData.TertiaryAxisDevice.PointingDevice
    };
    mouseEventsWebView.MouseEvent(mouseEvent);
}

[champnic]

I've converted this to a bug for the crash and added it to our backlog for the Hololens team to investigate.

[ArDevKarl]

Any insights about this issue? I'm running into the same problem