ScaleDetector doesn't work when a Component with DragCallbacks is added

[ojciec-dev]

Current bug behavior

ScaleDetector doesn't work when a Component with DragCallbacks is added.

Expected behavior

Scale gesture (pinch-to-zoom) should work regardless if a Component with DragCallbacks is registered. I'd expect to distinguish between two-finger scale gestures from one-finger tap and drag gestures.

In the example below you can drag the green rectangle but you cannot scale the board. When you remove DragCallbacks mixin, scale gesture starts working.

Steps to reproduce

1. Use ScaleDetector with FlutterGame
2. Add a child with DragCallbacks
3. Notice onScaleUpdate is not being called after scale gesture
4. Remove DragCallbacks mixin and notice onScaleUpdate is now being called

import 'package:flame/components.dart';
import 'package:flame/events.dart';
import 'package:flame/game.dart';
import 'package:flame/input.dart';
import 'package:flame/palette.dart';
import 'package:flutter/material.dart';

class ScaleDragGame extends StatelessWidget {
  const ScaleDragGame({super.key});

  @override
  Widget build(BuildContext context) {
    final game = ScaleDragFlameGame();

    return GameWidget(game: game);
  }
}

/// `onScale` methods are not called when a Component with DragCallbacks is added as a child.
/// Green rectangle can be dragged around but scale gestures don't work. But if I remove DragCallbacks from
/// [DraggableRectangle] scale gestures start to work.
class ScaleDragFlameGame extends FlameGame with ScaleDetector {
  late final CameraComponent cameraComponent;
  late double startZoom;

  ScaleDragFlameGame();

  @override
  Color backgroundColor() => Colors.yellow;

  @override
  Future<void> onLoad() async {
    final world = World();
    cameraComponent = CameraComponent(world: world)
      ..viewfinder.anchor = Anchor.center
      ..viewport.anchor = Anchor.center;
    addAll([world, cameraComponent]);

    world.add(DraggableRectangle(
      size: size / 4,
      position: size / 6,
      paint: BasicPalette.darkGreen.paint(),
    ));
  }

  @override
  void onScaleStart(ScaleStartInfo info) {
    startZoom = cameraComponent.viewfinder.zoom;
  }

  @override
  void onScaleUpdate(ScaleUpdateInfo info) {
    /// not called when Component with DragCallbacks was added

    final currentScale = info.scale.global;
    if (!currentScale.isIdentity()) {
      final newZoom = (startZoom * currentScale.y).clamp(1.0, 6.0);
      cameraComponent.viewfinder.zoom = newZoom;
    } else {
      final delta = info.delta.game;
      cameraComponent.viewfinder.position.translate(-delta.x, -delta.y);
    }
  }
}

/// Remove [DragCallbacks] and see that pinch-to-zoom gestures start working
class DraggableRectangle extends RectangleComponent with DragCallbacks {
  DraggableRectangle({
    super.position,
    super.size,
    super.paint,
  });

  @override
  void onDragUpdate(DragUpdateEvent event) {
    super.onDragUpdate(event);
    position += event.delta;
  }
}

Flutter doctor output

[✓] Flutter (Channel stable, 3.10.5, on macOS 13.2.1 22D68 darwin-arm64, locale en-US)
    • Flutter version 3.10.5 on channel stable at /Users/wojciechplesiak/Development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 796c8ef792 (7 weeks ago), 2023-06-13 15:51:02 -0700
    • Engine revision 45f6e00911
    • Dart version 3.0.5
    • DevTools version 2.23.1

More environment information

• Flame version: 1.8.1
• Platform affected: android
• Platform version affected: android 13

More information

I want to have draggable components placed on the zoomable board. Imagine you have a puzzle board and you want to zoom in to be able to drag the smaller pieces with precision. DragCallbacks is taking over all touch inputs even when they happen outside of the Component. I see under the hood that whenever a Component with DragCallbacks is added ImmediateMultiDragGestureRecognizer is registered as one of the gestureDetectors (here).

[spydon]

This is a really tricky limitation in Flutter, since scale is a superset of pan. We could possibly work around it by using the scale detector for panning too, but I don't think that multi touch would work properly if we did that. If anyone has a good solution for this we are all ears!

[ojciec-dev]

With regular Flutter widgets you can put Draggable inside InteractiveViewer and both drag and scale gestures work in the same time. I'm sure there is a way to do this and keep multitouch functional.

For now I was forced to stop using Flame for this very reason as I was unable to find a workaround on my end. What I tried was adding at first non-draggable Component, and then replacing it with a draggable Component after it was touched. Then on tapping outside - replacing draggable with non-draggable. This worked but only one time, looks like adding a Component with DragCallbacks will register a listener, but removing this component won't unregister that listener and this listener will steal the touch inputs after even though it's not doing anything. I also tried adding/removing listeners manually, but part of the Flame impl is internal and I couldn't do it.

[spydon]

I'm sure there is a way to do this and keep multitouch functional.

How would it be able to detect whether multiple fingers on the screen are moving independently or doing a scale gesture? Possibly we could have an implementation that supports one-finger drags and scaling at the same time, but I think it is basically impossible to support two finger scaling and multi-finger dragging at the same time since a lot of the drag events would register as scale events.

This worked but only one time, looks like adding a Component with DragCallbacks will register a listener

You can remove this listener by doing this I think:

game.findByKey(MultiDragDispatcherKey()).removeFromParent();

[ojciec-dev]

@spydon, I read my previous message and I see it's confusing, apologies

put Draggable inside InteractiveViewer and both drag and scale gestures work in the same time

what I meant is you can either drag a Draggable or scale with InteractiveViewer - depending what takes the touch input first, but you can perform both actions. My use case is being able to position elements on the screen with precision, and since elements can vary in size I need to be able to zoom in to be able to grab the smaller ones.

[spydon]

When you put a Draggable inside of an InteractiveViewer, you don't have multi touch support anymore, right? Because the scale events will always (or a lot of the time) win on the gesture arena as soon as there are multiple fingers involved. So what I'm suggesting is to create a new event dispatcher that registers a DragGestureRecognizer (monodrag) which should be possible to use together with a ScaleDetector/ScaleGestureRecognizer.

It should be fairly easy, mostly copying the current MultiDragDispatcher implementation and create something like MonoDragDispatcher and then add a boolean to DragCallbacks that indicates whether it should register a mono- or multi-dispatcher. And probably have an assertion if it finds out that there are different types trying to be registered at the same time.

[ojciec-dev]

No, you still get multitouch working, it just depends on what you click on first. From my experience:

1. when any Draggable widget gets the touch input first, scale gesture will not work, but pan would work
2. when scale gets the input first, Draggable won't receive any input

You can see it working here: https://github.com/flame-engine/flame/assets/46694136/5f4629d0-9c54-41fa-8e1e-b6583bb44ef8

Thanks for the hints how to implement similar behavior with Flame.

[spydon]

No, you still get multitouch working, it just depends on what you click on first. From my experience:

Interesting, there must be some really delicate work done in the gesture arena for that to work.

Thanks for the hints how to implement similar behavior with Flame.

If you want to implement it in Flame, do you want to make a PR with it? In that case i can assign you to the issue. :) You can of course ask me if you need any pointers too.

[ojciec-dev]

We moved away from Flame for now, I used at first to get the collision detection for free, but after I stumbled upon this issue I switched over to standard Flutter to be able to move forward. Down the road we might need some Flame magic, but util that happens I need to focus on other things.

[spydon]

Linking #2726 here since it is related.

[TecN01R]

Is there a workaround for this?

[osaxma]

I faced the opposite of the original issue -- i.e. DragCallbacks on a child component does not work properly with ScaleDetector as a parent component.

Details

In the video snippet below, you'll notice after zooming in then zooming out, dragging with a single finger invokes onScaleStart instead of onDragStart most of the time. In some cases, even tapping on on the game invokes onScaleStart.

https://github.com/flame-engine/flame/assets/46427323/f8e5ef77-7a94-4bc6-a2df-dcea0f767847

Minimal Code sample to reproduce:

Expand To View

import 'package:flame/components.dart';
import 'package:flame/events.dart';
import 'package:flame/game.dart';
import 'package:flame/input.dart';
import 'package:flutter/material.dart';

void main() {
  final game = ZoomExample(viewportResolution: Vector2(500, 500));
  runApp(
    MaterialApp(
      debugShowCheckedModeBanner: false,
      color: Colors.yellow,
      home: Center(
        child: Container(
          height: 500,
          width: 500,
          child: GameWidget(game: game),
        ),
      ),
    ),
  );
}

class ZoomExample extends FlameGame with ScrollDetector, ScaleDetector {
  ZoomExample({
    required Vector2 viewportResolution,
  }) : super(
          camera: CameraComponent.withFixedResolution(
            width: viewportResolution.x,
            height: viewportResolution.y,
          ),
        );

  @override
  Color backgroundColor() {
    // TODO: implement backgroundColor
    return Colors.blue;
  }

  @override
  Future<void> onLoad() async {
    world.add(SomeComponent());
  }

  void clampZoom() {
    camera.viewfinder.zoom = camera.viewfinder.zoom.clamp(0.05, 3.0);
  }

  static const zoomPerScrollUnit = 0.02;

  @override
  void onScroll(PointerScrollInfo info) {
    camera.viewfinder.zoom += info.scrollDelta.global.y.sign * zoomPerScrollUnit;
    clampZoom();
  }

  late double startZoom;

  @override
  void onScaleStart(_) {
    print('onScaleStart');
    startZoom = camera.viewfinder.zoom;
  }

  @override
  void onScaleUpdate(ScaleUpdateInfo info) {
    final currentScale = info.scale.global;
    if (!currentScale.isIdentity()) {
      camera.viewfinder.zoom = startZoom * currentScale.y;
      clampZoom();
    } else {
      final delta = info.delta.global;
      camera.viewfinder.position.translate(-delta.x, -delta.y);
    }
  }
}

class SomeComponent extends RectangleComponent with DragCallbacks {
  SomeComponent() : super(size: Vector2.all(50), position: Vector2.all(0));

  @override
  // TODO: implement paint
  Paint get paint => super.paint..color = Colors.red;

  @override
  void onDragStart(DragStartEvent event) {
    print('onDragStart');
    super.onDragStart(event);
  }

  @override
  void onDragUpdate(DragUpdateEvent event) {
    // TODO: implement onDragUpdate
    position += event.localDelta;
  }
}

Additional Info

For reference, I've already faced a similar issue with InteractiveViewer before and I've a bug filed in flutter repo: https://github.com/flutter/flutter/issues/136622

[osaxma]

Just to follow up:

The issue I reported above was a bug in the engine and it was fixed in:

• https://github.com/flutter/engine/pull/49424

Also, not sure if it'll help here, but the pointerCount for the trackpad was also updated:

Now trackpad gestures will count as pointerCount=2 instead of 1. It makes it easier for people who want to have different behaviour for single-finger drag vs two-finger pan/zoom.

• https://github.com/flutter/flutter/pull/140745

both fixes should be in master by now.

Cheers

[spydon]

Thanks for the update @osaxma, this should have a separate issue though since it's not the same as the one OP describes.