[web:multi-view] cannot scroll page when cursor lands on a FlutterView

[yjbanov]

When embedding a FlutterView onto a custom element, e.g. in multi-view mode, the app may have two sources of scrolling:

1. Flutter scrolls some content inside itself.
2. The host page scrolls its content, and one piece of content may be a FlutterView.

Case 1 seems to work well. However, case 2 does not work when the mouse cursor or finger (i.e. pointer) is on top of the FlutterView. When this happens the web engine calls preventDefault, which prevents browser scrolling. Somehow the engine needs to know when not to call preventDefault, but that information lives in the framework.

Demo: in the sample below, see that when the pointer is on top of the blue area the page is able to scroll, but if the pointer moved to the yellow area the scrolling no longer happens.

https://github.com/flutter/flutter/assets/211513/8b026e69-bff2-45df-b577-bd13f4a30c69

A couple of potential fixes:

• Change PointerDataPacketCallback to return some value to the engine that provides information about whether the pointer should be forwarded to the platform for handling.
• As part of rendering the frame, perhaps in the Scene graph, annotate regions with information about how certain pointer events should be handled.

My gut feeling tells me that the first option is better. For one, it is simpler and less invasive. But also, the logic that determines what to do with a pointer may depend on many things - kind, device, location, pressure - and the second option would have to pre-declare all this information even if a tiny subset of it is applicable.

This may require a collaboration between the framework and the web teams, but I'm assigning it to the framework team first to get guidance on what option would be the most idiomatic and work well across platforms.

[goderbauer]

Some thoughts about this:

Change PointerDataPacketCallback to return some value to the engine that provides information about whether the pointer should be forwarded to the platform for handling.

Based on a single pointer event (e.g. a pointer down) we cannot decide whether the framework will "handle" the event. For example, when we see a pointer down event delivered via the PointerDataPacketCallback this could be part of a tap gesture (which we want to handle in the framework) or it could end up being a scroll (which we may want to handle in the browser). We will only know for sure after we have seen a sequence of follow-up events and only at that point can we tell you whether the framework handled them or not (basically, once we know if any gesture handler claimed victory in the gesture arena). This information wouldn't be available synchronously when pointer events are delivered to the framework.

As part of rendering the frame, perhaps in the Scene graph, annotate regions with information about how certain pointer events should be handled.

I think this has the same problem, but in reverse: How are you on the engine-side going to predict that a pointer down event is going to be a scroll gesture (and needs to be handled directly without telling the framework) vs. its going to be a tap gesture that the framework needs to know about to activate a button?

Somehow, the gesture arena of the framework needs to work together with the gesture recognition system of the browser. Not sure how we can archive this, though, and if browsers have any APIs for this?

/cc @mdebbar

[flutter-triage-bot[bot]]

The triaged-web label is irrelevant if there is no team-web label or fyi-web label.

[goderbauer]

Earlier @yjbanov and I had a conversation about this and came up with two potential ideas:

Idea: Competition

We send a specially crafted member representing the browser into the gesture arena for every pointer. It can only win if it is the last remaining member and no other member wins. It remembers all the pointer events dispatched to it and the platform prevents defaults for all of them. When this special member "wins" the arena, we play the events back to the platform and let it handle it since nobody on the framework side wants them.

This will require some API on the platforms to inject events back into the platform. Such API may not exist.

Idea: Cooperation

For scroll wheel events we could check whether the PointerSignalResolver has any registered event handlers when PointerSignalResolver.resolve is called. If it has a registered handler for the event, then the event is handled by the framework and the platform is told synchronously to prevent defaults. If it has no registered handler, the platform needs to handle the event. This should basically resolve the problem altogether for scroll wheels because Scrollables only register a handler with the PointerSignalResolver if they can actually scroll in response to the event.

For scroll by touch we could extend GestureArenaMember with an option to prevent defaults when a particular member enters the gesture arena on a point down event. By default, all GestureDetectors would allow defaults on the GestureArenaMember that they send to the arena - except for the member added by a vertically scrolling Scrollable. Pointer events are now dispatched as usual, but when at the end of dispatching a pointer down event the arena is closed we check if any members disallow defaults. If that is the case, the platform is told to prevent defaults, otherwise it isn't. This isn't expected to solve the issue in all cases, though: If a scrollable is all the way at the top it can still scroll down and would therefore send a default-preventing member into the arena on pointer down to prevent defaults even when the pointer down event would be the start of a scroll up gesture that should be handled by the platform, but wouldn't be in this scenario. An argument could be made here that UIs with nested scrolling are kinda bad and should be avoided. Either you want the FlutterView to be scrollable or the outer page, but not both at the same time. Therefore, this could be a "good enough" solution that fulfills most use cases.

Both of these rely on the fact that upon receiving either a signal or a pointer down event from the platform we can synchronously message back to the platform whether defaults need to be prevented. In the framework, event handling is pretty much synchronous - with two known exceptions: locked events and resampling.

When events are locked incoming pointer events are queued and processed after events are unlocked again. Currently, we only lock events while producing a warmup frame and during hot reload. Hot reload isn't available yet on the web, so we may not have to worry too much about it. And even if it were, it would probably be fine if events arriving during this dev operation behave a little strange - after all, the app is reassembling itself. The same can be said about the warm-up frame right at app startup (and after hot reload).

If resampling is enabled (it is off by default) we may be out of luck since the whole point of it is to process event asynchronously. PointerSignalEvents appear to not be affected by resampling, so those could still work. It will need some additional investigation to see if PointerDown events are also dispatched asynchronously as part of resampling.

There may be further asynchronicity in event dispatch on the engine side. At least when semantics are enabled, the web engine seems to operate in async mode for event dispatch (see here). Further investigation is needed to figure out if we could make that work with this approach.

There are also the other non-web platforms to consider (supporting multi-view in an add-to-app fashion will surface the same problems there). They will likely involve a thread hop for event dispatch and further investigation will be needed to see if defaults can be prevented asynchronously for incoming pointer events.

[goderbauer]

Idea "competition" is kinda of how PlatformViews work today on at least iOS and (in the future) MacOS. It could likely also be used for view embedding on those platforms. (cc @cbracken)

@yjbanov How does the web do gesture disambiguation with platform views there to decide whether the platform view or the surrounding flutter app should handle it?

[mdebbar]

@goderbauer thanks for the write up!

When this special member "wins" the arena, we play the events back to the platform and let it handle it since nobody on the framework side wants them.

AFAIK, this is not possible on the web. We can create an event and dispatch it to the DOM, but the browser won't handle it.

For scroll by touch we could extend GestureArenaMember with an option to prevent defaults when a particular member enters the gesture arena on a point down event.

Note: after playing a bit with this in a jsfiddle, I found out the event that needs to be preventDefaulted is touchmove with {passive: false}. Preventing default on pointermove, for example, won't work.

How does the web do gesture disambiguation with platform views there to decide whether the platform view or the surrounding flutter app should handle it?

It is my understanding that on the web, platform views always swallow pointer events. There are a few reasons:

• In some cases, the browser doesn't even let us receive the events (e.g. if the platform view is an iframe).
• We can't replay events later if we decide to let the platform view handle them.

The solution we have right now is to let the app developer decide. If they want to let the platform view handle the events, then they don't have to do anything (this is the default behavior). But if they want to handle the events on the Flutter side, they can use pointer_interceptor.

[yjbanov]

Note: after playing a bit with this in a jsfiddle, I found out the event that needs to be preventDefaulted is touchmove with {passive: false}. Preventing default on pointermove, for example, won't work.

A-ha! That's why it didn't work in my experiments. Thanks for digging this up!

[flutter-triage-bot[bot]]

The triaged-web label is irrelevant if there is no team-web label or fyi-web label.

[commandiron]

any update?

[ditman]

I'm going to look at this next; focusing first on the wheel event, which for web desktop may cover 99% of the cases.

As for other events, one idea that I have (not sure if it can be done) is that the framework should tell the engine that it's ignoring a gesture, so the engine can stop forwarding events to the framework until they're needed again (for example if the framework tells us that it's ignoring some pointermove event, we can stop forwarding them until there's a new pointer up)

[ditman]

I have two RFC PRs to make wheel events from the browser "ignorable" by the framework, so the web engine conditionally calls preventDefault on them.

The current solution is focused on the Scroll event, but it could be extended to any "PointerSignal" (immediate events, not gestures).

• https://github.com/flutter/engine/pull/51566
• https://github.com/flutter/flutter/pull/145500

Demo:

• https://dit-tests.web.app

[goderbauer]

Demo: https://dit-tests.web.app/

This is very cool!

One thing I noticed: When I let my scroll wheel spin freely over a flutter view as soon as the scrollable in the flutter view has reached the bottom, the outside page starts scrolling without me ever touching the scroll wheel again. On native web nested scrolling, this doesn't seem to be the case. The outside page only starts scrolling when I am at the bottom of the inner scrollable and a new scroll wheel sequence is started. The events from the same scroll initiation are not shared between the two scrollables.

[goderbauer]

I have two RFC PRs to make wheel events from the browser "ignorable" by the framework, so the web engine conditionally calls preventDefault on them.

Does this also work for two-finger touchpad scrolling? Or only scrolling with an actual physical mouse wheel?

Edit: confirmed that this also works for touch pad scrolling with the same somewhat jaring effect described above that as soon as the end of the flutter scrollable is reached, the remaining inertia of the gesture is used to scroll the outside page.

[ditman]

On native web nested scrolling, this doesn't seem to be the case.

Good feedback! I did test the touchpad on my mac, but haven't tried to fling the page.

The current implementation could be refined in the web engine by having a "lock" once the framework starts handling "wheel" events by which we preventDefault of all remaining "wheel" events until something else releases the lock (like a "pointerdown" or "pointermove").

I don't know if there's a better way of telling apart normal wheel events from those emitted by the remaining inertia (maybe comparing its delta* values)

[The-RootCause]

Any way this can becomes a P1 and gets addressed ? this is blocking #69529 which is making the flutter web unusable. Thank you

[ditman]

@The-RootCause my implementation is not fixing #69529, it's just dealing with wheel scroll events. Gesture recognition is slightly different.

[Garoth]

OK after much suffering with "embedded flutter inside other html (hugo) mobile scrolling", using this thread, I have the following solution. The general idea is to sidestep everything by having the first click onto the app "fullscreen" the app and move into flutter mode. There's an overlaid HTML button to exit this. In the background, scroll events and various details are manipulated, like selectively disabling touchmove:

<script>
    function allowDragScrollingMobileFlutterEmbed(e) {
        e.preventDefault();
    };

    window.addEventListener('load', function(ev) {
        // Control pointer events to flutter, since it eats scroll
        var body = document.querySelector('body');
        var flutterDiv = document.querySelector('#flutter');
        var parent = flutterDiv.parentElement;
        var flutterExit = document.querySelector('#flutter-exit');
        parent.addEventListener('click', function() {
            flutterDiv.classList.add('clicked');
            flutterExit.classList.add('visible');
            body.classList.add('unscrollable');
            document.addEventListener('touchmove',
                allowDragScrollingMobileFlutterEmbed, { passive: false });
        });
        flutterExit.addEventListener('click', function(e) {
            flutterDiv.classList.remove('clicked');
            flutterExit.classList.remove('visible');
            body.classList.remove('unscrollable');
            document.removeEventListener('touchmove',
                allowDragScrollingMobileFlutterEmbed, { passive: false });
            e.stopPropagation();
            e.preventDefault();
        });

        // Downloads main.dart.js
        console.log('[FlutterInit] serviceWorkerVersion:', serviceWorkerVersion);
        _flutter.loader.loadEntrypoint({
            serviceWorker: {
                serviceWorkerVersion: serviceWorkerVersion,
            },
            onEntrypointLoaded: function(engineInitializer) {
                engineInitializer.initializeEngine({
                    hostElement: document.querySelector("#flutter")
                }).then(function(appRunner) {
                    appRunner.runApp().then(function(){
                        flutterDiv.classList.add('visible');
                    });
                });
            }
        });
    });
</script>

That's obviously missing CSS and HTML, but perhaps this idea will help someone out

[ditman]

@Garoth full-screening the Flutter view is a cool feature!

[ditman]

@Garoth looking at your code, I think this PR may be relevant to you:

• https://github.com/flutter/engine/pull/53647

[yjbanov]

Scrolling using the mouse and the touchpad has landed, which was in the MVP of the feature. Touch-based scrolling can be addressed as a follow-up.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.