View tracking not a as single ray

[arianaa30]

Puting it simple, is there any ways to define gaze as a box, say a rectangle that might intersect with multiple objects? Basically I want to detect whatever objects are currently within user's view and user can see; not necessarily a single point as the end of a ray. With a single point I can't really achieve that.

[NolaDonato]

Are you interested in obtaining the list of currently visible objects, everything the user can see? Or do you want to further constrain it to a smaller gaze box, a subset of what is visible?

[arianaa30]

I want to experiment both! I understand we have 96 degree of view in GearVR, so for a box it can be a 70ish degree or sth maybe.

[arianaa30]

Any thoughts on this? It is a bit urgent for me to achieve this.

[NolaDonato]

I can add something to GVRPicker to pick against the camera's view frustum. It will return the list of currently visible objects. Will that work?

[arianaa30]

Sure. It would be best to make it as a configurable box (given a height and width as degrees?!) centered on the gaze. And pick whatever is within/intersects with that. If it is too much work, just visibility works.

[NolaDonato]

Pull request #765 adds GVRPicker.pickVisible which will return the list of visible objects. Please test this pull request and see if it does what you want. If so we can do further refinement.

[arianaa30]

Thanks. Just that I'm not very familiar with the pull request. I went to your repository and downloaded the whole 'pickvisible' branch. Then I replaced the common files in my current Framework with that of the downloaded ones (but didn't remove the existing ones), but when I am trying to compile it, it has errors! What should I do? Aren't there any plans to update the master branch of Gearvrf so I can download the whole thing from scratch or something?!

[arianaa30]

OK, I replaced only the changed GVRPicker files and fixed an error due to my API being a bit older and things compiled Ok. I was using the pickVisible() in a similar way as pickObjects(), but it is not working, and in my example, the non-visible objects get selected! The selection also lags a lot, and blinks. Is it fully tested to make sure it works, or maybe my test is problematic?

I do this in onStep() : My objects turn into objectsRed if they are NOT visible.

        for (GVRPickedObject pickedObject : GVRPicker.pickVisible(mGVRContext.getMainScene())) {
            for (int i = 0; i < objectsRed.length; i++) {
                if (pickedObject.getHitObject().equals(objectsRed[i])) {
                    if (!objects[i].isEnabled()) {
                        objectsRed[i].setEnable(false);
                        objects[i].setEnable(true);
                        break;
                    }
                } else if (!objectsRed[i].isEnabled()) {
                    objects[i].setEnable(false);
                    objectsRed[i].setEnable(true);
                    break;
                }
            }

    }

[NolaDonato]

Here is how to get the pull request into your own local repository: git checkout -b pr768 master git pull https://github.com/NolaDonato/GearVRf.git x3dverts

The first line makes a new branch based on the Framework master. The second line pulls over my changes into that branch. I am not sure if just copying the files will work.

How many objects do you have? I will try and make a test like yours and see if mine lags too.

[arianaa30]

I dont have many objects. Just 4 of objects and 4 of the objectsRed. They are video objects in my real test. Can you please test with my onStep() code to make see if it works?

[rahul27]

I have a simple solution for your problem.

Since you already know all the objects in your scene, all you need to do is to create a cube scene object and attach it to the camera rig. You then use the isColliding() call to detect a collision. I have modified the simple sample, to show you how it's done:

https://github.com/rahul27/GearVRf-Demos/commit/6aa4e1bdc26f3386c3c89a8c1c91c13b70a7c0a5

isColliding() is relatively inexpensive since it is a simple box-box test.

You will need to use the setPosition call appropriately to make sure that the cube is placed at an appropriate depth to ensure collision. For eg. the quad in the sample is at a depth of 3.0f, we therefore use:

cubeSceneObject.getTransform().setPosition(0.0f, 0.0f, -3.0f);

[NolaDonato]

This will not clip to a frustum. It will report false positives where the back corners of the cube are out of view.

[arianaa30]

So does Rahul's approach work?! I din't understand it why using cubeSceneObject can help. Anyways If it fully works without false positives, let's test it. Let me know.

[danke-sra]

For an approximation, you can only test bounding boxes to be viewable -- this is, it's projection in camera frame falls in the specified gaze box -- similar to culling. If you care about pixel-level accuracy, you may need to test all triangles in the mesh (which takes time if not optimized).

[rahul27]

@arianaa30 from the sounds of it your requirements seem simple. It works as long as you don't have a pixel level requirement like @danke-sra mentioned.

[NolaDonato]

I updated the pull request to generate pick events for frustum picking. I added a new class GVRFrustumPicker which will return the list of colliders that fall within the camera view frustum. It works just like GVRPicker generating onEnter, onExit and OnInside events. If you look at the gvr-eyepicking sample and replace GVRPicker with GVRFrustumPicker you should get events for objects as they move in and out of the view frustum. I am working on making this more general to allow picking from any viewpoint but that is not working yet so don't call GVRFrustumPicker.setFrustum - it is documented but not functional.

[arianaa30]

OK. So basically instead of GVRPicker.pickVisible(mGVRContext.getMainScene()), I have to just another class called GVRFrustumPicker, right? It has pickObjects() inside?

[arianaa30]

One thing is that currently I have 4 object. I might have 6, 10, ... so just wanted to make sure the cubeSceneObject works independently of number of objects.

[arianaa30]

Fortunately I was able to compile Framework, again, by replacing the modified files, which is good. Shall I still use pickObjects function as before or pickVisible?

[rahul27]

If you are using the GVRFrustumPicker then pickVisible should work.

[NolaDonato]

No, Rahul suggested I change it to work with events and I did. You will need to reapply the pull request because I have updated it with new code. Then you can use the picking events like the gvr-eyepicking sample but use GVRFrustumPicker instead of GVRPicker.

From: arianaa30 [mailto:[email protected]] Sent: Thursday, August 4, 2016 11:47 AM To: Samsung/GearVRf [email protected] Cc: Nola Donato [email protected]; Comment [email protected] Subject: Re: [Samsung/GearVRf] View tracking not a as single ray (#762)

Fortunately I was able to compile Framework, again, by replacing the modified files, which is good. Shall I still use pickObjects function as before or pickVisible?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Samsung/GearVRf/issues/762#issuecomment-237646679, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQw1S17pv8IT-OXoI6nXL7P76krLiUmXks5qcjO3gaJpZM4JY9GI.

[arianaa30]

It doesn't work for me. I'm guessing probably something is wrong with the logic of my code. I have video objects here.

onStep(){
        for (GVRPickedObject pickedObject : GVRFrustumPicker.pickVisible(scene)) {
            for (int i = 0; i < videolow.length; i++) {
                if (pickedObject.getHitObject().equals(videolow[i])) {
                    if (!video[i].isEnabled()) {
                        videolow[i].setEnable(false);
                        video[i].setEnable(true);
                        break;
                    } 
                } else if (!videolow[i].isEnabled()) {
                    video[i].setEnable(false);
                    videolow[i].setEnable(true);
                    break;
                }
            }
        }
}

[arianaa30]

Yes, my code has problems. While I fix that, any solutions for picking whatever intersects with front box of X degrees (say 60 degrees) as opposed to the whole screen which is visible?

[rahul27]

@arianaa30 are you still having problems? I just tested this pull request https://github.com/Samsung/GearVRf/pull/765 and it worked fine for me.

You could try changing the following line from GVRPicker to GVRFrustumPicker to test:

https://github.com/gearvrf/GearVRf-Demos/blob/master/gvr-eyepicking/app/src/main/java/org/gearvrf/gvreyepicking/SampleMain.java#L73

Regarding your code, it would be difficult to understand what could be going wrong by just looking at your snippet. I would add logs to onStep() to check if the correct objects are picked:

Log.d(TAG, "Picked Object " + pickedObject.getName());

Also I am assuming that your have break statements in your code for a reason.

[arianaa30]

I thought maybe my snippet is slow, so I changed my snippet and used hashMaps instead, but nothing changed.

So looking at the link, is it a new way of doing things?! Previously I was attaching a eyepicking to objects and had a method (got from your eyePicking example): attachDefaultEyePointee(object); and

    private void attachDefaultEyePointee(GVRSceneObject sceneObject) {
        sceneObject.attachEyePointeeHolder();
    }

Now for this one I should just attach it to the scene using these two lines?!

        mainScene.getEventReceiver().addListener(mPickHandler);
        mainScene.getMainCameraRig().getOwnerObject().attachComponent(new GVRPicker(gvrContext, mainScene));

[rahul27]

Yup. That is a new events based picking model the framework. You should look at the sample to see how it works.

Also @NolaDonato has been kind enough to make a video that describes it:

https://youtu.be/7xk0mva6Rzk

You could also look at the developer guide:

http://gearvrf.org/bin/view/GearVRfDeveloperGuide/Picking

[NolaDonato]

The attachEyePointeeHolder function is a special case of attachCollider which attached a component to your scene object that makes it pickable. You have to attach a collider to every scene object you are going to pick. Then you should attach the listener for the pick events and a GVRFrustumPicker to generate the events.

Do this for all the scene objects you want to be pickable:

private void attachDefaultEyePointee(GVRSceneObject sceneObject) {

    sceneObject.attachEyePointeeHolder();

}

Do this in your onInit to make the picker generate events and to handle them:

mainScene.getEventReceiver().addListener(mPickHandler);

    mainScene.getMainCameraRig().getOwnerObject().attachComponent(new GVRFrustumPicker(gvrContext, mainScene));

[NolaDonato]

Did you get the frustum picker to work for you? Do you need me to send you a full sample?

[arianaa30]

I didn't have a chance yet- so bisy yesterday. I will try to test out today. Thanks a lot for your followups!

[NolaDonato]

I updated pull request #765 to let you specify the view frustum to use for picking in GVRFrustumPicker. You can call GVRFrustumPicker.setFrustum(float fovy, float aspect, float near, float far) to set the viewing area to be different from the camera. Let me know if that works for you. I tested it by changing the following lines in the gvr-eyepicking sample: // remove this line in onInit // mainScene.getMainCameraRig().getOwnerObject().attachComponent(new GVRPicker(gvrContext, mainScene));

// add these lines in onInit float znear = mainCamera.getNearClippingDistance(); float zfar = mainCamera.getFarClippingDistance(); GVRSceneObject headTransform = mainCamera.getHeadTransform().getOwnerObject(); GVRFrustumPicker picker = new GVRFrustumPicker(gvrContext, mainScene); picker.setFrustum(40.0f, 1.0f, znear, zfar); headTransform.attachComponent(picker);