Can't view HandRigging plugin mesh on Android mobile

[mairead]

I'm trying to connect a second device using the Leapjs-network plugin. I've got the data being piped over WebRTC and I can see the Hand mesh moving if I view in two separate connected desktop browsers but I can't see the Hand mesh appearing on a mobile browser.

I was wondering if you'd successfully managed to connect a mobile device and see the Hand mesh moving. It would be good to clarify if you think this should be possible and if you could tell me what device and browsers you managed to connect successfully. Its hard to tell if I have a hardware or software limitation.

I have a Three.Js texture error in Chrome for Android, which may be a separate issue. The texture error varies depending on which version of Three.js I'm using (Either S3TC or OES) but Firefox for Android looks like its working OK and throws no rendering errors.

I'm using ADB remote debugging for Firefox on Android and I get the following error messages

Error: WebGL: Exceeded 2 live WebGL contexts for this principal, losing the least recently used one. (From Three.js)

TypeError: this is undefined

    this.remoteFrameLoop = function() {

        var frame, frameData;
        if (this.controller.streaming()) {// <--here
          return;
        }

(From the LeapJs-network plugin) inside the remoteFrameLoop, but I think thats probably just caused by the first frame where my hand is not present so no data is available

and a couple of deprecation warnings from Peer.Js about adding the IceCandidate which I think probably aren't important

Would really appreciate any advice you might have. I think I've hit a dead end here

[raimo]

Hi @mairead and thanks for using leapjs-network! I haven't tried the mobile issue your talking about, but maybe the plugins you're using is adding WebGL contexts than you need? Are you loading the rigged-hand plugin, for instance? A bit of code would help tracking down the TypeError a bit better.

[mairead]

Hey Raimo, So I have two application servers. One is running an application that the Leap is plugged into

    window.controller = new Leap.Controller({
      background: true,
      checkVersion: false
    });
    controller.use('networking', {
      peer: new Peer({key: 'vg930sy60kck57b9'}),
      plotter: new LeapDataPlotter()
    })
    controller.use('riggedHand', {
    });
    controller.connect();
    var peer = controller.plugins.networking.peer;
    document.getElementById("bigred").onclick = function(){
        var remoteId = document.getElementById("connectionId").value;
        controller.plugins.networking.connect(remoteId);
        peer.on('connection', function(conn){
            console.log("new connection", conn)
        });
    }
    controller.on('frame', function(frame) {
        var remoteId = document.getElementById("connectionId").value;
    });

The second is pulling in the incoming data and trying to animate the rigged hand on a mobile browser. I can see the frame data getting passed over. I just can't see the rigged Hand mesh appearing on the screen.

    window.controller = new Leap.Controller({
          background: true,
          checkVersion: false
    });
    controller.use('networking', {
        peer: new Peer({key: 'vg930sy60kck57b9'})
    });
    controller.use('riggedHand', {

    });
    var RemoteApp = new Peer('remoteApp', {key: 'vg930sy60kck57b9'}); 
    var peer = controller.plugins.networking.peer;
    peer.on('connection', function(conn){
        conn.on('data', function(data){
            //console.log(data)
            //outputDebugging.showLeapData(data.frameData);
        });
    });

I'm using the following Leap plugins: protocol-events, plugins, networking, riggedHand and Leap data plotter and the Peer Js lib and Three.js v70, although I've swapped in a few of the latest versions of Three and that doesn't seem to resolve the issue.

Ultimately what I'd like to do is pass the frame data over into a separate Three.js VR application which is taking the accelerometer data from the mobile device and so I can use accelerometer data and leap frame data in the same WebGL context to render the Three.js screen.

The only way to get the Leap data onto the browser on the mobile device is to use websockets or webRTC which is how I came across the Leapjs-networking lib. I did have an earlier prototype using plain websockets to pipe the frame data from the websockets straight into the browser but I think the streamed data had a slightly different footprint and I was wondering if that is the function of the frameSplicer and frame Packer in this plugin. To repackage data sent over a socket and then reconstruct in a way the riggedHand can utilise

At the moment it would appear that the mobile device isn't receiving the frame data to make the Hand appear.

[mairead]

If it helps I can stick up a demo app somewhere or put the repos on github?

[mairead]

So I've been looking at this a bit more this afternoon. It would appear as if the issue in Firefox is caused by the return window.requestAnimationFrame(frameSplicer.remoteFrameLoop); not setting up a repeating loop, so I'm guessing the HandRigging never has a chance to kick in because the frame rendering isn't taking place.

https://github.com/leapmotion/leapjs-network/blob/master/build/leap.networking-0.1.0.js#L116

In Chrome you can see the requestAnimationFrame getting called repeatedly, but in Firefox it stops after it reaches this is undefined.

Am I right in thinking remoteFrameLoop function is initially called in the setTimeout after a connection is opened?

What I see is the FrameSplicer.remoteFrameLoop is called once and the value of this is the FrameSplicer and then its called a second time and this is undefined.

In Chrome I either get the FrameSplicer object as the value of this inside the loop, or what looks like the window object. I've attached a couple of screenshots of my console windows.

[raimo]

You're probably right there.

However, something that I would try to look into is controller.use('riggedHand', { }) and passing it the Three.js canvas/context, because it builds its own AFAIK if you don't connect your existing canvas to it:

https://github.com/leapmotion/leapjs-rigged-hand/blob/master/src/leap.rigged-hand.coffee#L249

[mairead]

Hi, I just wanted to let you know I finally got something working. I came across a note in the rigged hand docs, saying if you set the scene you also have to set the renderer, so its possible I might not have been passing in both. Also I switched out a reference in the network plugin to this for the actual frameSplicer object reference to get rid of the Firefox issue.

     this.remoteFrameLoop = function() {
            //console.log("this when called by requestAnimationFrame", this)
            //console.log("framespliver called by requestAnimationFrame", this)
            var frame, frameData;
            // replace this with object 
            if (frameSplicer.controller.streaming()) {
            // if (this.controller.streaming()) {
              return;
            }
            frameData = {
              hands: [],
              pointables: []
            };
            frameSplicer.addRemoteFrameData(frameData);
            frame = new Leap.Frame(frameData);
            frameSplicer.supplementFinishedFrame(frame, frameData);
            // replace this with object 
            frameSplicer.controller.processFrame(frame);
            //this.controller.processFrame(frame);
            //console.log('value of frameSplicer ',frameSplicer)
            return window.requestAnimationFrame(frameSplicer.remoteFrameLoop);
          };

Works in Android chrome and Firefox, so I guess the THREE.WebGLRenderer: OES_texture_float_linear extension not supported error is a red herring.

[mairead]

I've come across another issue which is intermittent but I thought I'd mention it in case anyone else is having the same issue. My mobile device occassionally has trouble connecting. I've isolated it down to this line in FrameSplicer.prototype.addRemoteFrameData

if ((new Date).getTime() > (remoteFrame.sentAt + this.options.frozenHandTimeout)

It seems as if the remoteFrame.sentAt timestamp is sometimes really far behind the current timestamp which means the device won't display the frameDate being passed across.

I guess it could be network latency. I've definitely noticed that the motion on a mobile device suffers latency issues over time, whereas connecting another browser on my desktop seems to not to.

I've put a really gross hack in to bump up the timeout value for now to force the rigging to appear but I'd be keen to know if other people have the same issue and whether they've manage to resolve it

[raimo]

Thank you for the note! Excited to hear how your mobile project goes.

[mairead]

Hi Julia,

I can't work out where to reply to you on github so I hope you get this message.

The node servers aren't running on the mobile device. I have a server running on my local desktop machine and then I'm pointing a web browser to the node server running the client side hand plugin.

For some reason the leap data recorded from the USB and piped into the leap websocket isn't available to the mobile device browser. I don't know why this is. their documentation suggests you should be able to point it to any IP but my application would only work if the data was coming into the localhost and then I'm using a bit of a hack with the web RTC to pipe the data to another server.

Ideally this step shouldn't be required but I spent a lot of time trying to force my mobile device to connect to the Leap socket before eventually going down the RTC route. This might be my mistake though I'm not suggesting this is the only/best solution.

On Sat, Aug 8, 2015 at 11:00 PM, Julia Jacobs [email protected] wrote:

Just curious @mairead https://github.com/mairead : did you need to actually run a node server on your android device to do this? I'm trying to do get leapmotion to work with my iphone in google cardboard and was hoping i didn't actually have to run a node emulator to do this but be able to just point either the mobile ios v8 safari or chrome browsers to regular separate web server. Thanks!

— Reply to this email directly or view it on GitHub https://github.com/leapmotion/leapjs-network/issues/2#issuecomment-129054871 .

[jewelsjacobs]

Sorry I removed my comment because I figured you didn't do this.

I appreciate the response. I’m having the same issue. I was not able to get the leapmotion data from my laptop by pointing to my laptop IP either.

I jailbroke my iPhone and am able to install a jailbreak nodeJS hack and ssh into the filesystem.

I’m going to see if I can first:

Move https://gist.github.com/jewelsjacobs/114f579c9e72dad57ec5#file-leapmotion-network-html to the iPhone and open it in the safari mobile browser using a jailbreak hack that lets you open files in mobile safari while also running it on the laptop.

If that doesn’t work, actually try running grunt serve off of the iPhone directly.

If you have any code I can poach that you feel might help or any additional information it would be very much appreciated!

Thanks!

[jdonald]

@mairead has already gotten the rigged hand rendering in Chrome for Android, so my suggestions here may not be essential. Nonetheless, here are a few general hints for anyone attempting to use the leapjs rigged hand on Android/iOS.

• If using the Leap Motion Android SDK to connect the device via the OTG port, see the README in the Google Drive folder. For https + secure WebSocket, I now include instructions for installing a localhost certificate starting with the openssl x509 -inform PEM -text -in cert.crt ... command, as well as caveats when testing insecure WebSocket.
• If reading the WebSocket output directly from a desktop server's IP address, be sure to run leapd/LeapSvc with --websockets_allow_remote. However, if you are proxying the data such that it would appear to come from localhost on the mobile device, this flag is probably not necessary.
• We have not tested on iOS with Safari, but since there does not exist a native iOS Leap Motion SDK that runs on the phone/tablet, the above bullet regarding --websockets_allow_remote on the server likely applies.

[mairead]

Hi, sorry if I'm slow to reply. I am reading the comments but i have a 10 day old baby at home so, even when I can get to the computer i can only type one handed.

Setting the remote flag didnt work for me but again that might be user error on my part.

@jewelsjacobs You can take a look at my repos here. They don't actually need to be in two separate Node servers but I was messing around and its ended up that way. Essentially my app is one client side app connecting to another over web rtc so they could both be served from the same Node backend.

https://github.com/mairead/leapserver

https://github.com/mairead/three-d-hand

Point the mobile browser to the port that three-d-hand is running on. You can run these both on the desktop too to check its actually working,

I've got some slides here which might give you a bit of background too, although I know slides without notes can be a bit pointless. There is video showing you what the app is supposed to look like though

http://www.emdeebeebee.com/webrebels