Issues with loading OBJ file into TextureMesh and visualizing it

[taketwo]

Recently I've bumped into issues trying to display a textured mesh in visualizer. The mesh was comping from a .obj file produced by MVS-Texturing. The visualized geometry was alright, but the textures were messed up. I remembered that we have a bunch of issues related to this. I did a small research and tried some quick fixes, but the problem turned out to be more complex. I don't have time to address it entirely now, but since I've already invested some time, I decided to summarize my findings for future reference. If someone wants to have this issue fixed and has a reasonable spare time budget, welcome! I will be willing to guide and provide assistance.

Reports

GitHub

Specifically this problem:

• TextureMesh struct needs additional texture index buffer public attribute (#1592)
• loadOBJFile() cannot load texture (#1602)
• Problem in loadOBJFile (#1169)
• OBJReader() wrong texture from MTL file (#1615)

(I will close all of them in favor of this umbrella issue.)

Related:

• Exporting textureMesh (#1966)

Elsewhere

• pcl-users
• stack overflow

Problem

According to my understanding, the problem is two-fold and needs fixes in multiple PCL modules.

Texture coordinate indices

The first part of the problem has been reported in #1592. In a nutshell, the TextureMesh class lacks texture coordinates indices and the loader ignores such indices (if they are present in an .obj file). Refer to the original report for more details.

Hidden assumption in PCLVisualizer::addTextureMesh()

The second part of the problem has to do with how TextureMesh is displayed with PCLVisualizer. According to my understanding, the hidden assumption in the code is that there should be one-to-one correspondence between vertices (i.e. points in the TextureMesh.cloud) and UV coordinates in TextureMesh.tex_coordinates. This means that same vertex can not have different UV coordinates when it appears in different faces.

Quick hack

A quick and dirty solution (proposed in #1169) is to load the geometry and texture coordinates with loadPolygonFileOBJ (i.e. VTK-based reader) and materials with loadOBJFile (i.e. PCL's own loader) and then merge them:

pcl::TextureMesh mesh1;
pcl::io::loadPolygonFileOBJ ("mesh.obj", mesh1);
pcl::TextureMesh mesh2;
pcl::io::loadOBJFile ("mesh.obj", mesh2);
mesh1.tex_materials = mesh2.tex_materials;
pcl::visualization::PCLVisualizer visu("Test");
visu.addTextureMesh (mesh1,"texture");
visu.spin ();

This works for a single material case, but I suspect will fail if there are multiple materials. (But I did not test.) The reason why this works is that the VTK reader duplicates vertices and texture coordinates for each face.

Solution plan

1. As proposed in #1592, introduce texture coordinate indices in TextureMesh.
2. Update the OBJ file loader to populate these indices
   • If .obj file has texture indices read them;
   • If .obj file does not have texture indices generate sequential indices to mimic current behavior.
3. Update PCLVisualizer to use the texture coordinate indices. This means that every vertex of every face should be added to the vtkPolyData points array (duplicating as needed), same with texture coordinates.
4. (?) Update TextureMapping class. I haven't looked into that, but since it produces TextureMesh objects it probably needs to be updated to populate the newly introduced member field.
5. (?) Double-check that loaders from other formats (e.g. STL) still produce valid TextureMeshes.

To verify the results and avoid regressions, we would also need to collect a set of .obj files for testing. This should include files produced both by PCL and other software; with and without texture indices; with single and multiple materials.

Ideally, the changes in the loader class should be supported by unit tests (which are currently non-existent).

Another approach

VTK 6.3 has a new vtkOBJImporter class (merge request) that can handle materials. We could piggy-back on it. However:

• Ubuntu 16.04, which will still be around for a while has VTK 6.2;
• Ubuntu 18.04, which will be released next month, is going to have VTK 6.3, whereas the upstream version is 8.1. The inital version of the importer had numerous bugs that were fixed over time (examine the commit history).

[stale[bot]]

Marking this as stale due to 30 days of inactivity. It will be closed in 7 days if no further activity occurs.

[EwingKang]

This issue is still there. Just add to the discussion, when calling pcl::io::loadPolygonFileOBJ(), an integer return value is defined. But there are no definitions for it. I print it and get a return value of 149997, which matches the vertices count.

[INFO] [1707277875.882010659] [mesh_publisher]: Loaded mesh with 24999 points. [INFO] [1707277876.118122107] [mesh_publisher]: Loaded textured mesh msg with 149997 points with 49999 triangles and a texture map of 1313467 bytes. ret: 149997

Maybe the documentation needs some update, too. Since one could presume a return of 0 on, success < 0 on error as many other io functions do in the documents.

[mvieth]

@EwingKang Which PCL version do you use? Have you tried pcl::io::loadOBJFile instead? Can you describe your problem in more detail, please? What are you trying to do and what happens instead? I will take a look at the documentation of pcl::io::loadPolygonFileOBJ and fix it when I have the time.

[EwingKang]

@mvieth Thanks for the reply! We're having the Lunar New Year currently so I'm afraid I can't provide more details. I was using the version that came with apt on a Ubuntu 22.04. Not sure if it's the version that came with ros-pcl or just standalone PCL. l'll provide more info maybe next week once we get back to the job.

[mvieth]

@EwingKang Sure, no problem. On Ubuntu 22.04, apt would give you PCL 1.12.1. This problem is fixed in PCL 1.13.0 and newer. So using a newer PCL version should solve the issue for you.

[EwingKang]

Just to follow up: On my system I'm using PCL 1.12.1 as @mvieth has mentioned. So I presume it's not a problem anymore.

libpcl-apps1.12/jammy,now 1.12.1+dfsg-3build1 amd64 [installed,automatic] Point Cloud Library - apps library

libpcl-common1.12/jammy,now 1.12.1+dfsg-3build1 amd64 [installed,automatic] Point Cloud Library - common library

My original issue was that, when load an .obj mesh file with a single mesh using pcl::io::loadOBJFile, the size of loaded tex_coordinates[0] is 0. However, as stated in this issue, loadPolygonFileOBJ doesn't fill-in the tex_materials fields. So I followed the Quick hack solution mentioned by the issue owner, and the workaround worked.

I really appreciate the super fast response and accurate information. I would try the newer version sometime. Thanks again for mvieth's help.