Android ArCore OpenGL world to screen coordinates

[arusyakhov]

This is not a bug but a question, which I wasn't able to mark with a question mark. Any answer and help will be much appreciated

In a project of augmented face tracking I need to get screen coordinates for face specific point having mesh vertices, mesh normals. I know the solution with sceneform sdk, but I don't want to use it, as it has a bug on Samsung Galaxy S9 (face is detected with diversion to left and up). I try to get screen coordinates of augmented face point this way, but get wrong results

            val x1 = face.meshVertices.get(FACE_UPPER_CENTRAL_POINT * 3)
            val y1 = face.meshVertices.get(FACE_UPPER_CENTRAL_POINT * 3 + 1)
            val z1 = face.meshVertices.get(FACE_UPPER_CENTRAL_POINT * 3 + 2)
            val x2 = face.meshVertices.get(FACE_LOWER_CENTRAL_POINT * 3)
            val y2 = face.meshVertices.get(FACE_LOWER_CENTRAL_POINT * 3 + 1)
            val z2 = face.meshVertices.get(FACE_LOWER_CENTRAL_POINT * 3 + 2)

            val point1 = getScreenCoordinates(Vector3(x1, y1, z1), viewMatrix, projectionMatrix)
            val point2 = getScreenCoordinates(Vector3(x2, y2, z2), viewMatrix, projectionMatrix)

fun getScreenCoordinates(coord: Vector3, viewMatrix: FloatArray, projectionMatrix: FloatArray): Vector2 {

        val view = IntArray(4)
        view[0] = 0
        view[1] = 0
        view[2] = App.helper.getDisplayWidth()
        view[3] = App.helper.getDisplayHeight()
        val pos = FloatArray(4)
        GLU.gluProject(coord.x, coord.y, coord.z, viewMatrix, 0, projectionMatrix, 0, view, 0 , pos, 0)


        return Vector2(pos[0], pos[1])
    }

[devbridie]

Are you including the face's center pose in the calculation? Keep in mind that meshVertices coordinates are described relative to the face's center pose.

[arusyakhov]

Hi @devbridie . Thank you for your response. No, my calculation includes mesh vertices (x,y, z) values (written in my question). Could you please tell how should face center pose be included in the calculation?

[devbridie]

You could use something like this:

operator fun Vector3.plus(pose: Pose) = Vector3(x + pose.tx(), y + pose.ty(), z+pose.tz())

val woordCoordinatesPosition = Vector3(x1, y1, z1) + face.centerPose 

[arusyakhov]

Thanks again for the response @devbridie . I get the world coordinates with your suggestion and then convert them to screen coordinates via

fun getScreenCoordinates(coord: Vector3, viewMatrix: FloatArray, projectionMatrix: FloatArray): Vector2 {

    val view = IntArray(4)
    view[0] = 0
    view[1] = 0
    view[2] = getDisplayWidth()
    view[3] = getDisplayHeight()
    val pos = FloatArray(4)
    GLU.gluProject(coord.x, coord.y, coord.z, viewMatrix, 0, projectionMatrix, 0, view, 0 , pos, 0)


    return Vector2(pos[0], pos[1])
}

where

        val projectionMatrix = FloatArray(16)
        camera.getProjectionMatrix(projectionMatrix, 0, 0.1f, 100.0f)

        val viewMatrix = FloatArray(16)
        camera.getViewMatrix(viewMatrix, 0)

But I still get inaccurate coordinates. Do you have any suggestions or can you help with if I am doing anything wrong?

[MaxZGit]

Hi, is it possible to place an Object that is rotated so that it faces towards a real-world location. I try to build a signpost which should point in the right direction. I am currently using the sample-project and want to modify it as mentioned above.