Is it possible to know which UV pixel point would be a 3d vertex position on a 3d model?

[daqjjang]

Hello, I am trying to figure out which UV pixel point(2d) would be a 3d position on a 3d model. Is it possible? (The 3d position to a pixel point is also okay.) I know three pixel points for the vertex points on the triangular mesh, but I want to know the UV pixel values for points inside the triangle.

[Kiord]

Yes, it is possible.

UV -> 3D

write a position map :

def compute_interpolation_map(shape, tcoords, values):
    points = (tcoords * np.asarray(shape)[None, :]).astype(np.int32)
    x = np.arange(shape[0])
    y = np.flip(np.arange(shape[1]))
    X, Y = np.meshgrid(x, y)
    res = griddata(points, values, (X, Y),  method='cubic')
    res[np.isnan(res)] = 0
    return res

mesh = trimesh.load(path, process=False)
image_size = (512, 512)
position_map = compute_interpolation_map(image_size, mesh.visual.uv, mesh.vertices)

Note that compute_interpolation_map lets you create a map by linearly interpolate any per-vertex value (e.g. normal vector)

3D -> UV

mesh = trimesh.load(path, process=False)

# Get the corresponding mesh triangles

tids = trimesh.proximity.closest_point(mesh, points)[2]

# Get the barycentric coordinates of the points in their respective triangle

def barycentric_coordinates_3d(vertices, points):
    v0 = vertices[:, 1, :] - vertices[:, 0, :]
    v1 = vertices[:, 2, :] - vertices[:, 0, :]
    v2 = points - vertices[:, 0, :]
    d00 = np.einsum('ij,ij->i', v0, v0)
    d01 = np.einsum('ij,ij->i', v0, v1)
    d11 = np.einsum('ij,ij->i', v1, v1)
    d20 = np.einsum('ij,ij->i', v2, v0)
    d21 = np.einsum('ij,ij->i', v2, v1)
    denom = d00 * d11 - d01 * d01
    barys = np.zeros_like(points)
    barys[:, 1] = (d11 * d20 - d01 * d21) / denom
    barys[:, 2] = (d00 * d21 - d01 * d20) / denom
    barys[:, 0] = 1.0 - barys[:, 1] - barys[:, 2]
    return barys

#barys = barycentric_coordinates_3d(mesh.vertices[mesh.faces[tids]], points)
barys = trimesh.triangles.points_to_barycentric(mesh.vertices[mesh.faces[tids]], points, method='cramer') # almost 3 times faster !
 
# Finally, the uv coordinates

uvs = np.einsum('ij,ijk->ik', barys, mesh.visual.uv[mesh.faces[tids]])

I did not test anything but it should work

[mikedh]

Thanks for the writeup @Kiord!

Just wondering, does the above barycentric_coordinates_3d perform better than trimesh.triangles.points_to_barycentric?

https://github.com/mikedh/trimesh/blob/52fded0ada210358dacef8a8b1de82d71615ade6/trimesh/triangles.py#L468-L497

[Kiord]

Hello @mikedh, thanks for all the great work !

I should have searched a bit more in Trimesh because I did not know this function was already implemented. I made a simple benchmark :

trimesh.triangles.points_to_barycentric(method='cramer'): 2.128999 ms for 100 tests.
trimesh.triangles.points_to_barycentric(method='cross') : 4.126702999999999 ms for 100 tests.
barycentric_coordinates_3d : 5.631712999999996 ms for 100 tests.

barycentric_coordinates_3d is the slowest of the 3 methods so @daqjjang should use trimesh.triangles.points_to_barycentric with method='cramer'.

Thank you for pointing this out !

To go back to the original question

The first UV->3D solution that I proposed is not optimal if you have only a few queries to do or if you need to get precise 3D positions (they will be limited by the image resolution).

Instead, another solution is to use the barycentric coordinates (as mentioned in the 3D->UV solution):

tids = trimesh.proximity.closest_point_uv(mesh, uv) 
barys = trimesh.triangles.uv_to_barycentric(mesh.visual.uv[mesh.faces[tids]], uv)
points = np.einsum('ij,ijk->ik', barys, mesh.vertices[mesh.faces[tids]])

AFAIK, these functions are not implemented in Trimesh and a PR could be made. I doubt that it is very useful though.

EDIT We would need :

• An attribute triangles_uv_tree similar to triangles_tree but with 2d coordinates in trimesh.Trimesh class.
• A function trimesh.proximity.nearby_faces_uv similar to trimesh.proximity.nearby_faces
• A function trimesh.proximity.closest_point_uv similar to trimesh.proximity.closest_point
• A function trimesh.triangles.uv_to_barycentric similar to trimesh.triangles.points_to_barycentric
• A lot of code can be factorized between the "uv" version and "3d" version of these.

[daqjjang]

Thank you so much!

[nguyenquocduongqnu]

Yes, it is possible.

UV -> 3D

write a position map :

def compute_interpolation_map(shape, tcoords, values):
    points = (tcoords * np.asarray(shape)[None, :]).astype(np.int32)
    x = np.arange(shape[0])
    y = np.flip(np.arange(shape[1]))
    X, Y = np.meshgrid(x, y)
    res = griddata(points, values, (X, Y),  method='cubic')
    res[np.isnan(res)] = 0
    return res

mesh = trimesh.load(path, process=False)
image_size = (512, 512)
position_map = compute_interpolation_map(image_size, mesh.visual.uv, mesh.vertices)

Note that compute_interpolation_map lets you create a map by linearly interpolate any per-vertex value (e.g. normal vector)

3D -> UV

mesh = trimesh.load(path, process=False)

# Get the corresponding mesh triangles

tids = trimesh.proximity.closest_point(mesh, points)[2]

# Get the barycentric coordinates of the points in their respective triangle

def barycentric_coordinates_3d(vertices, points):
    v0 = vertices[:, 1, :] - vertices[:, 0, :]
    v1 = vertices[:, 2, :] - vertices[:, 0, :]
    v2 = points - vertices[:, 0, :]
    d00 = np.einsum('ij,ij->i', v0, v0)
    d01 = np.einsum('ij,ij->i', v0, v1)
    d11 = np.einsum('ij,ij->i', v1, v1)
    d20 = np.einsum('ij,ij->i', v2, v0)
    d21 = np.einsum('ij,ij->i', v2, v1)
    denom = d00 * d11 - d01 * d01
    barys = np.zeros_like(points)
    barys[:, 1] = (d11 * d20 - d01 * d21) / denom
    barys[:, 2] = (d00 * d21 - d01 * d20) / denom
    barys[:, 0] = 1.0 - barys[:, 1] - barys[:, 2]
    return barys

#barys = barycentric_coordinates_3d(mesh.vertices[mesh.faces[tids]], points)
barys = trimesh.triangles.points_to_barycentric(mesh.vertices[mesh.faces[tids]], points, method='cramer') # almost 3 times faster !
 
# Finally, the uv coordinates

uvs = np.einsum('ij,ijk->ik', barys, mesh.visual.uv[mesh.faces[tids]])

I did not test anything but it should work

How to save uv coordinates (uvs) as an image? Thanks.

[mha142]

An errorr pops up from using "mesh.visual.uv" the error is: AttributeError: 'ColorVisuals' object has no attribute 'uv'

[math-sasso]

How can I get those points from this line?

tids = trimesh.proximity.closest_point(mesh, points)[2]

I assume those are not the vertices from

mesh = trimesh.Trimesh(
                vertices=vertices_numpy,
                faces=faces_numpy,
                vertex_colors=color_numpy,
            )