Wrong frame passed to Gesture event emitter

[jo3w4rd]

[Edited and retitled with a more precise description of the problem]

After adding a gesture event callback to a Controller object, the callback is invoked multiple times for the same event/frame. (Only happens with swipes and circles; taps seem fine.) What appears to be happening is that the frame passed to the onGesture callback is the latest frame, not the frame to which the gesture object belongs. This makes it difficult to access finger positions, etc. at the time of the gesture update.

var controller = new Leap.Controller({enableGestures:true});
controller.connect();

controller.on('gesture', onGesture);
function onGesture(gesture,frame)
{
    console.log(gesture.type + " with ID " + gesture.id + " in frame " + frame.id);
}

Expected behavior is that the frame passed to the callback is the frame in which a specific gesture update occurred -- not the latest frame.

[0x-r4bbit]

@jo3w4rd Yea @robinboehm and me having the same problem in our angular.js adapter. However, kinda great to see that it's not an issue on our site. We're currently working around this with a timeout blocker. Hopefully this gets fixed very soonish!

[jo3w4rd]

@PascalPrecht , @robinboehm, Are you sure this is the same problem? Swipes are tied to fingers, not the hand, so you could potentially have 5 swipes generated simultaneously with one movement of the hand. Gestures from the Leap should be considered raw data -- you still need to massage them into something reasonable for your application (as compared to gestures from the OS or browser -- which have already been massaged for you).

[0x-r4bbit]

@jo3w4rd Ah! Good catch! Didn't know that! We'll investigate.

[robinboehm]

@jo3w4rd Documentation says: "Swipe — A linear movement of the hand." - that's a little bit confusion :) I've tested it with "finger-swipe" and it works really good. Thx for the tip! :)

[pehrlich]

Just ran a test, and this appears to be working fine. Perhaps you are not checking the gesture state, which may be start, stop, or update?

    console.log( gesture.state + ' ' + gesture.type + " with ID " + gesture.id + " in frame " + frame.id);

Here's a 5 finger swipe test case

    start swipe with ID 7 in frame 43876 
    start swipe with ID 49 in frame 43876 
    start swipe with ID 54 in frame 43876 
    update swipe with ID 7 in frame 43877 
    update swipe with ID 49 in frame 43877 
    update swipe with ID 54 in frame 43877 
    start swipe with ID 51 in frame 43877 
    start swipe with ID 4 in frame 43878 
    update swipe with ID 7 in frame 43878 
    update swipe with ID 49 in frame 43878 
    update swipe with ID 54 in frame 43878 
    update swipe with ID 51 in frame 43878 
    update swipe with ID 4 in frame 43879 
    update swipe with ID 7 in frame 43879 
    update swipe with ID 49 in frame 43879 
    update swipe with ID 54 in frame 43879 
    update swipe with ID 51 in frame 43879 
    update swipe with ID 4 in frame 43880 
    update swipe with ID 7 in frame 43880 
    update swipe with ID 49 in frame 43880 
    update swipe with ID 54 in frame 43880 
    update swipe with ID 51 in frame 43880 
    update swipe with ID 4 in frame 43881 
    update swipe with ID 7 in frame 43881 
    update swipe with ID 49 in frame 43881 
    update swipe with ID 54 in frame 43881 
    update swipe with ID 51 in frame 43881 
    update swipe with ID 4 in frame 43882 
    update swipe with ID 7 in frame 43882 
    update swipe with ID 49 in frame 43882 
    update swipe with ID 54 in frame 43882 
    update swipe with ID 51 in frame 43882 
    update swipe with ID 4 in frame 43883 
    update swipe with ID 7 in frame 43883 
    update swipe with ID 49 in frame 43883 
    update swipe with ID 54 in frame 43883 
    update swipe with ID 51 in frame 43883 
    update swipe with ID 4 in frame 43884 
    update swipe with ID 7 in frame 43884 
    update swipe with ID 49 in frame 43884 
    update swipe with ID 54 in frame 43884 
    update swipe with ID 51 in frame 43884 
    update swipe with ID 4 in frame 43885 
    update swipe with ID 7 in frame 43885 
    update swipe with ID 49 in frame 43885 
    update swipe with ID 54 in frame 43885 
    update swipe with ID 51 in frame 43885 
    update swipe with ID 4 in frame 43886 
    update swipe with ID 7 in frame 43886 
    update swipe with ID 49 in frame 43886 
    update swipe with ID 54 in frame 43886 
    update swipe with ID 51 in frame 43886 
    update swipe with ID 4 in frame 43887 
    update swipe with ID 7 in frame 43887 
    update swipe with ID 49 in frame 43887 
    update swipe with ID 54 in frame 43887 
    update swipe with ID 51 in frame 43887 
    update swipe with ID 4 in frame 43888 
    update swipe with ID 7 in frame 43888 
    update swipe with ID 49 in frame 43888 
    update swipe with ID 54 in frame 43888 
    update swipe with ID 51 in frame 43888 
    update swipe with ID 4 in frame 43889 
    update swipe with ID 7 in frame 43889 
    update swipe with ID 49 in frame 43889 
    update swipe with ID 54 in frame 43889 
    update swipe with ID 51 in frame 43889 
    update swipe with ID 4 in frame 43890 
    update swipe with ID 7 in frame 43890 
    update swipe with ID 49 in frame 43890 
    update swipe with ID 54 in frame 43890 
    update swipe with ID 51 in frame 43890 
    update swipe with ID 4 in frame 43891 
    update swipe with ID 7 in frame 43891 
    update swipe with ID 49 in frame 43891 
    update swipe with ID 54 in frame 43891 
    update swipe with ID 51 in frame 43891 
    update swipe with ID 4 in frame 43892 
    update swipe with ID 7 in frame 43892 
    update swipe with ID 49 in frame 43892 
    update swipe with ID 54 in frame 43892 
    update swipe with ID 51 in frame 43892 
    update swipe with ID 4 in frame 43893 
    update swipe with ID 7 in frame 43893 
    update swipe with ID 49 in frame 43893 
    update swipe with ID 54 in frame 43893 
    update swipe with ID 51 in frame 43893 
    update swipe with ID 4 in frame 43894 
    update swipe with ID 7 in frame 43894 
    update swipe with ID 49 in frame 43894 
    update swipe with ID 54 in frame 43894 
    update swipe with ID 51 in frame 43894 
    update swipe with ID 4 in frame 43895 
    update swipe with ID 7 in frame 43895 
    update swipe with ID 49 in frame 43895 
    update swipe with ID 54 in frame 43895 
    update swipe with ID 51 in frame 43895 
    update swipe with ID 4 in frame 43896 
    update swipe with ID 7 in frame 43896 
    update swipe with ID 49 in frame 43896 
    update swipe with ID 54 in frame 43896 
    update swipe with ID 51 in frame 43896 
    update swipe with ID 4 in frame 43897 
    update swipe with ID 7 in frame 43897 
    update swipe with ID 49 in frame 43897 
    update swipe with ID 54 in frame 43897 
    update swipe with ID 51 in frame 43897 
    update swipe with ID 4 in frame 43898 
    update swipe with ID 7 in frame 43898 
    update swipe with ID 49 in frame 43898 
    update swipe with ID 54 in frame 43898 
    update swipe with ID 51 in frame 43898 
    update swipe with ID 4 in frame 43899 
    update swipe with ID 7 in frame 43899 
    update swipe with ID 49 in frame 43899 
    update swipe with ID 54 in frame 43899 
    update swipe with ID 51 in frame 43899 
    update swipe with ID 4 in frame 43900 
    update swipe with ID 7 in frame 43900 
    stop swipe with ID 49 in frame 43900 
    stop swipe with ID 54 in frame 43900 
    stop swipe with ID 51 in frame 43900 
    stop swipe with ID 4 in frame 43901 
    stop swipe with ID 7 in frame 43901 

The gestures are accumulated from deviceFrames on to animationFrames, but for update events especially, which happen on an animationFrame basis, there's no reason we'd need to attach the deviceFrame info instead of animationFrame info.

Please reopen this if I've misunderstood!

[jo3w4rd]

The central issue, which can occur with the animationFrame loop, is that there is information in the Gesture object which is tied to a frame (Pointable ID, Hand IDs), but a different frame object is passed to the gesture event callback.

Thus if you try to look up the pointable or hand in the frame by id, and then use the values, then you will get the wrong information. In fact the hand or pointable may no longer even exist.

[pehrlich]

Hmm. Have you had this occur regularly in some use case?

With some basic hand-waving, and this sample code, it hasn't been an issue:

  ctl.on('frame', function(frame){
      if (frame.gestures.length && frame.gestures[0].pointableIds.length){
        for (var i =0; i < frame.gestures[0].pointableIds.length; i++){
          if (!frame.pointable(frame.gestures[0].pointableIds[i])) {
            debugger;
          }
        }
      }
    });

But sure, its possible. A couple possible fixes:

• Include frameId in gesture. That can then be used to look up the frame in the history.
• Gesture could also include a frame() method, which would do that, or also finger(), pointable(), etc.

EDIT: Looks like that can't be done so easily -- history only holds animationFrames in the browser (https://github.com/leapmotion/leapjs/blob/master/lib/controller.js#L168)

[jo3w4rd]

We may just have to document that there is a potential mismatch between the gesture and the current frame when using animationFrames.

[pehrlich]

We talked it over in the JS meeting today - seems like the frame history should really be holding deviceFrames, not animationFrames (which I believe is currently the case). When we make the switch, then we'll be able to pop a frameid on to the gesture, and have the data be much more accessible.

But yes for now (that's estimated 0.5.0)-- documentation.