texture may be very blurred in some area after mesh decimation

[CanCanZeng]

Hi, I use OpenMVS on ubuntu16.04 to create textured mesh. The original result is rather good, but I find that if I downsample the mesh using OpenMVS or other algorithm, some area may become very blurred. Is there any way to relieve this effect?

original: TextureMesh -i v2.0_mesh.mvs -o v2.0_mesh_texture.mvs --export-type ply and the result looks like below:

decimated by openMVS: TextureMesh -i v2.0_mesh.mvs -o v2.0_mesh_texture_decimated.mvs --export-type ply --decimate 0.1 and the result looks like below:

decimated by other algorithm:

The texture of the generated mesh retains a lot of detail in some places and is very blurry in others, which is very confusing.

[cdcseacave]

indeed, maybe you decimate the mesh to much and the triangles become too big (hard to see without a wireframe rendering) you can also try to decimate the mesh after texturing, using for ex Meshlab which has decimation for textured meshes

[CanCanZeng]

the mesh with wireframe rendering is showing below: original:

decimated by openMVS:

decimated by other algorithm:

The reason I downsampled the mesh is that the density of original triangles is too high and the texture process is very slow. So I can't downsample after the texture is done.

[cdcseacave]

0.1 is too much, the trinagles are too big for texturing

[CanCanZeng]

OK, I know decimate to 0.1 is too much.

But I still no understand why the mesh is only blurred in some area, is there any way to make the blur more evenly?

[CanCanZeng]

I found the causer! It is caused by outlier rejection process. Because the patch is too large, the average color cannot be used for outlier rejection, so the above problem is solved if I turn off outlier rejection. But turn off outlier rejection brings other problems, the textures are very strange in places with reflections.

In fact we have a dense point cloud, the color of the point cloud is very close to the texture color of the mesh, can we use the color or visibility of dense point cloud as a clue?

[cdcseacave]

can you post some screenshots here?

[CanCanZeng]

OK

close outlier rejection: the texture on the wall looks very good, but the texture on the floor is bad.

open outlier rejection: the texture on the wall is what I have posted above(some place may blurred very much), but the texture on the floor is right.

[cdcseacave]

I see indeed, we need to think at a new way of filtering the outlier views per face...

[CanCanZeng]

What about color histogram or gradiant histogram? Say a gray-scale color histogram with 16 bins.

[cdcseacave]

yes, that should be much more robust, and probably not much slower I'd start though with an 8 bin histogram, each bin on 8 bin counter pls help with the implementation if you can

[CanCanZeng]

OK, I'm also trying to implement it

[cdcseacave]

great, thank you, pls poke me if I can help with any advice/info

[CanCanZeng]

Hi, I realized a simple one, the result is not perfect yet, may be you can help me enhance it. I made a pull request here https://github.com/cdcseacave/openMVS/pull/829

I use color histogram instead of intensity histogram right now, since I find that the preformance of using intensity histogram is rather poor. And the performance now will be better if the mesh is more sparse, using very dense mesh will decrease the quality of texture, since too few pixels in one patch, the histogram can not represent the patch very well.

And I use correlation coefficient to measure the similarity of two patch, there must be better ways, any suggestion is welcome!

[cdcseacave]

I think this is a good start. Some thoughts:

• I think intensity histogram should work, maybe something else is the problem here
• i'd replace the offset-like correlation with a pre histogram equalization: https://en.wikipedia.org/wiki/Histogram_equalization
• some type of correlation is ok, you can use ZNCC for ex
• indeed, small triangles will be a problem, take a look at this PR https://github.com/cdcseacave/openMVS/pull/807 which combine small adjacent faces into larger virtual faces before texturing, you should try your idea here

[CanCanZeng]

i'd replace the offset-like correlation with a pre histogram equalization: https://en.wikipedia.org/wiki/Histogram_equalization

What does it mean? Do histogram equalization one the color image or the patch histogram? I find that the histogram of a patch tends to be very sparse, only one or two bins has data, so apply histogram equalization directly may not be possible.

some type of correlation is ok, you can use ZNCC for ex

I just replace NCC with ZNCC, the result does not have much difference.

indeed, small triangles will be a problem, take a look at this PR https://github.com/cdcseacave/openMVS/pull/807 which combine small adjacent faces into larger virtual faces before texturing, you should try your idea here

I'll try it later.

[cdcseacave]

histogram equalization on the patch intensities histogram

[CanCanZeng]

I have a problem using MVS::PointCloud::Octree, the code is listed below:

// construct octree for point cloud
MVS::PointCloud::Octree octPoints(scene.pointcloud.points, [](MVS::PointCloud::Octree::IDX_TYPE size, MVS::PointCloud::Octree::Type /*radius*/) {
    return size > 512;
});

// search in a small sphere
PointCloud::Point center = (pts[0] + pts[1] + pts[2]) / 3;
float radius = std::max(std::max(cv::norm(pts[0] - center), cv::norm(pts[1] - center)), cv::norm(pts[2] - center));

// search dense points in this sphere
MVS::PointCloud::Octree::IDXARR_TYPE point_indices;
octPoints.Collect(point_indices, center, radius);

the last line will cause error when compilation,

openMVS/libs/Common/Octree.inl:380: error: no matching function for call to ‘SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::_Collect(const SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::CELL_TYPE&, SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::AABB_TYPE, SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::IndexInserter) const’
  380 |  _Collect(m_root, AABB_TYPE(center, radius), IndexInserter(indices));
      |  ^~~~~~~~

I did not find any example on using Collect function in that way, so I have no idea how to fix it , can you help?

[cdcseacave]

you are using it correctly, I've just tested your code on my OpenMVS project and it works

[CanCanZeng]

That is so wired! I use it in SceneTexture.cpp, maybe I need to include some header files?

[CanCanZeng]

I compile OpenMVS on ubuntu20.04, gcc=9.4.0, intel i7-8700k cpu. the full error log is:

[ 15%] Built target Common
[ 22%] Built target Math
[ 42%] Built target IO
Scanning dependencies of target MVS
[ 45%] Building CXX object libs/MVS/CMakeFiles/MVS.dir/PointCloud.cpp.o
[ 45%] Building CXX object libs/MVS/CMakeFiles/MVS.dir/SceneTexture.cpp.o
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/MVS/SceneTexture.cpp: In member function ‘bool MeshTexture::ListCameraFaces(MeshTexture::FaceDataViewArr&, const IIndexArr&) const’:
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/MVS/SceneTexture.cpp:835:14: warning: unused variable ‘points’ [-Wunused-variable]
  835 |  const auto& points = scene.pointcloud.points;
      |              ^~~~~~
In file included from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.h:233,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Types.h:2796,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Common.h:176,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/MVS/Common.h:42,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/build/Debug/libs/MVS/CMakeFiles/MVS.dir/cmake_pch.hxx:5,
                 from <command-line>:
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.inl: In instantiation of ‘void SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::Collect(SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::IDXARR_TYPE&, const POINT_TYPE&, TYPE) const [with ITEMARR_TYPE = SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>; TYPE = float; int DIMS = 3; DATA_TYPE = unsigned int; SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::IDXARR_TYPE = SEACAVE::cList<long unsigned int, long unsigned int, 0, 1024, long unsigned int>; typename ITEMARR_TYPE::IDX = long unsigned int; SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::POINT_TYPE = Eigen::Matrix<float, 3, 1>]’:
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/MVS/SceneTexture.cpp:878:56:   required from here
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.inl:380:2: error: no matching function for call to ‘SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::_Collect(const SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::CELL_TYPE&, SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::AABB_TYPE, SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::IndexInserter) const’
  380 |  _Collect(m_root, AABB_TYPE(center, radius), IndexInserter(indices));
      |  ^~~~~~~~
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.inl:294:6: note: candidate: ‘void SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::_Collect(const SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::CELL_TYPE&, const AABB_TYPE&, INSERTER&) const [with INSERTER = SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::IndexInserter; ITEMARR_TYPE = SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>; TYPE = float; int DIMS = 3; DATA_TYPE = unsigned int; SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::AABB_TYPE = SEACAVE::TAABB<float, 3>]’ <near match>
  294 | void TOctree<ITEMARR_TYPE,TYPE,DIMS,DATA_TYPE>::_Collect(const CELL_TYPE& cell, const AABB_TYPE& aabb, INSERTER& inserter) const
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.inl:294:6: note:   conversion of argument 3 would be ill-formed:
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.inl:380:2: error: cannot bind non-const lvalue reference of type ‘SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::IndexInserter&’ to an rvalue of type ‘SEACAVE::TOctree<SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>, float, 3>::IndexInserter’
  380 |  _Collect(m_root, AABB_TYPE(center, radius), IndexInserter(indices));
      |  ^~~~~~~~
In file included from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Types.h:2796,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Common.h:176,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/MVS/Common.h:42,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/build/Debug/libs/MVS/CMakeFiles/MVS.dir/cmake_pch.hxx:5,
                 from <command-line>:
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.h:184:7: note: candidate: ‘template<class INSERTER, class COLLECTOR> void SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::_Collect(const SEACAVE::TOctree<ITEMARR_TYPE, TYPE, DIMS, DATA_TYPE>::CELL_TYPE&, TYPE, const COLLECTOR&, INSERTER&) const [with INSERTER = INSERTER; COLLECTOR = COLLECTOR; ITEMARR_TYPE = SEACAVE::cList<SEACAVE::TPoint3<float>, const SEACAVE::TPoint3<float>&, 0, 8192>; TYPE = float; int DIMS = 3; DATA_TYPE = unsigned int]’
  184 |  void _Collect(const CELL_TYPE&, TYPE, const COLLECTOR&, INSERTER&) const;
      |       ^~~~~~~~
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.h:184:7: note:   template argument deduction/substitution failed:
In file included from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.h:233,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Types.h:2796,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Common.h:176,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/MVS/Common.h:42,
                 from /media/zcc/data/Project/Work/DenseReconstruction/openMVS/build/Debug/libs/MVS/CMakeFiles/MVS.dir/cmake_pch.hxx:5,
                 from <command-line>:
/media/zcc/data/Project/Work/DenseReconstruction/openMVS/libs/Common/Octree.inl:380:2: note:   candidate expects 4 arguments, 3 provided
  380 |  _Collect(m_root, AABB_TYPE(center, radius), IndexInserter(indices));
      |  ^~~~~~~~
make[2]: *** [libs/MVS/CMakeFiles/MVS.dir/build.make:289: libs/MVS/CMakeFiles/MVS.dir/SceneTexture.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:493: libs/MVS/CMakeFiles/MVS.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

[cdcseacave]

this looks like some crazy GCC problems with lvalues (I use Windows); try replacing:

 _Collect(m_root, AABB_TYPE(center, radius), IndexInserter(indices));

with

IndexInserter inserter(indices);
 _Collect(m_root, AABB_TYPE(center, radius), inserter);

[CanCanZeng]

Wow, that fix the compilation problem!

[CanCanZeng]

I got the best texture result ever on my data by using the covisibility information of dense point cloud! I will post the code after I clean it up.

[CanCanZeng]

Hi @cdcseacave , can you try my code on windows? I find that my code runs well on ubuntu16.04 with gcc5, and the debug version also runs well on ubuntu20.04 with gcc9, but the release version will crash. I guess it also ascribes to the optimization of GCC9, can you help me check it?

[CanCanZeng]

Sorry, I forgot one thing: to use covisibility of dense point cloud, there should be dense point cloud and views info in .mvs file, but the default scene_mesh.mvs file does not have these infos, so there are two ways to use my code.

1. comment out this line https://github.com/cdcseacave/openMVS/blob/f34a258862af967456b7c4578aa2d8821875eb25/libs/MVS/SceneReconstruct.cpp#L876 to preserve point cloud info in output scene_mesh.mvs file
2. use scene.mvs but provide mesh file from command line, like below: TextureMesh -i scene.mvs --mesh-file scene_mesh.ply

[Yzhbuaa]

@CanCanZeng Hello, I don't understand that why the point cloud visibility info would help to detect outliers. Cause it seems there is not so mush difference between mesh visibility and point cloud visibility. Could you explain that in detail? Thank you!

[CanCanZeng]

The visibility of the point cloud is obtained in the MVS stage, and it is impossible for occluders to participate in the generation of the point cloud. The visibility of the mesh is obtained by projecting the mesh onto the image, so occluders cannot be excluded.