Endless loop in cmod::Mesh::pick()

[375gnu]

With large meshes (like cartrite's 3dmars) cmod::Mesh::pick(const Eigen::Vector3d&, const Eigen::Vector3d&, cmod::Mesh::PickResult*) goes into endless loop.

[UnknowableCoder]

Sorry, I was looking through this repo just so see if I could contribute to something and I spotted this bug and wanted to see if I could help. Took a look at the code in celmodel/mesh.cpp, is there not a problem with i0 not being updated when primType == TriFan?

In lines 432 to 441, we have:

                else // primType == TriFan
                {
                    index += 1;
                    if (index < nIndices)
                    {
                        index += 1;
                        i1 = i2;
                        i2 = group->indices[index];
                    }
                }

Shouldn't the second index += 1; (line 437) be some update of i0?

I'm not entirely sure of all the specifics of the mesh storage formats, but it seems unreasonable to my (ignorant) understanding that i0 does not change in this codepath, and the increment of index on the condition index < nIndices seems like an error (if index == nIndices - 1, we'd end up accessing group->indices[nIndices]).

[375gnu]

This image shows Triangle Fan. A is i0, B, C, D, etc are i1 and i2. So i0 is common and no need to update it.

and the increment of index on the condition index < nIndices seems like an error (if index == nIndices - 1, we'd end up accessing group->indices[nIndices]

Yes, this code is very fragile and potentially can cause UB.

[UnknowableCoder]

Thank you very much for the explanation and sorry for any inconvenience...