OpenCV 3D Mesher Visualiser is broken with OpenCV 4.2.0

[Wallbraker]

Description:

Command:

./scripts/stereoVIOEuroc.bash

Console output:

$ ./scripts/stereoVIOEuroc.bash 
Using dataset at path: /home/jakob/Kimera/euroc
 Launching:

            ██╗  ██╗██╗███╗   ███╗███████╗██████╗  █████╗
            ██║ ██╔╝██║████╗ ████║██╔════╝██╔══██╗██╔══██╗
            █████╔╝ ██║██╔████╔██║█████╗  ██████╔╝███████║
            ██╔═██╗ ██║██║╚██╔╝██║██╔══╝  ██╔══██╗██╔══██║
            ██║  ██╗██║██║ ╚═╝ ██║███████╗██║  ██║██║  ██║
            ╚═╝  ╚═╝╚═╝╚═╝     ╚═╝╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝

 
I1124 15:24:22.512121  6540 EurocDataProvider.cpp:82] Parsing Euroc dataset...
I1124 15:24:22.657095  6540 RegularVioBackEnd.cpp:114] Using Regular VIO backend.
I1124 15:24:22.663285  6540 PipelineModule.h:449] MISO Pipeline Module: Display has no output queue registered.
I1124 15:24:22.663506  6540 Pipeline.cpp:471] Pipeline Modules launched (parallel_run set to 1).
I1124 15:24:22.663684  6577 Pipeline.h:199] Spinning Kimera-VIO.
I1124 15:24:22.663718  6578 Pipeline.cpp:389] Shutting down VIO pipeline once processing has finished.
I1124 15:24:22.663753  6578 Pipeline.cpp:402] Statistics
-----------                                  #	Log Hz	{avg     +- std    }	[min,max]
Data Provider [ms]                      	    0	
Display [ms]                            	    0	
Mesher [ms]                             	    0	
VioBackEnd [ms]                         	    0	
VioFrontEnd [ms]                        	    0	
Visualizer [ms]                         	    0	
backend_input_queue Size [#]            	    0	
data_provider_left_frame_queue Size [#] 	    0	
data_provider_right_frame_queue Size [#]	    0	
display_input_queue Size [#]            	    0	
mesher_backend Size [#]                 	    0	
mesher_frontend Size [#]                	    0	
stereo_frontend_input_queue Size [#]    	    0	
visualizer_backend_queue Size [#]       	    0	
visualizer_frontend_queue Size [#]      	    0	
visualizer_mesher_queue Size [#]        	    0	
I1124 15:24:22.673558  6576 EurocDataProvider.cpp:115] Running dataset between frame 50 and frame 2000
W1124 15:24:22.732393  6574 Mesher.cpp:1380] Missing landmark information to build 3D Mesh.
W1124 15:24:22.733566  6575 OpenCvVisualizer3D.cpp:479] No landmark information for Visualizer. Not displaying 3D points.
W1124 15:24:22.741331  6540 OpenCvDisplay.cpp:130] Missing Mesh in visualization's 3D widgets.
W1124 15:24:22.794688  6574 Mesher.cpp:1380] Missing landmark information to build 3D Mesh.
terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(4.2.0) ../modules/core/src/matrix_expressions.cpp:23: error: (-5:Bad argument) Matrix operand is an empty matrix. in function 'checkOperandsExist'

Additional files: Please attach all the files needed to reproduce the error.

Please give also the following information:

• Kimera-Vio branch, tag or commit used: master Merge pull request #128 from MIT-SPARK/feature/updated_realsense_params
• GTSAM version used: master
• OpenGV version used: master
• OpenCV version used: 4.2.0
• Operating system and version (e.g. Ubuntu 16.04 or Windows 10): Ubuntu 20.04
• Did you change the source code? (yes / no):

[Pallav1299]

I am also facing the same issue on Ubuntu 20.04.

[duncanrhamill]

Same problem on Pop (Ubuntu) 20.10 with OpenCV 3.4.12. Seems to come from OpenCvVisualizer3d::visualizeMesh3D getting empty color and tcoord matrices, changing

src/visualizer/OpenCvVisualizer3D.cpp:623

  // No points/mesh to visualize.
  if (map_points_3d.rows == 0 || polygons_mesh.rows == 0) {
    return;
  }

to

  // No points/mesh to visualize.
  if (map_points_3d.rows == 0 
    || polygons_mesh.rows == 0 
    || colors.rows == 0 
    || tcoords.rows == 0
  ) {
    return;
  }

solved the problem for me.

[Wallbraker]

The above fix doesn't show any mesh, applying this patch allows me to show uncoloured mesh.

diff --git a/src/visualizer/OpenCvVisualizer3D.cpp b/src/visualizer/OpenCvVisualizer3D.cpp
index bab1f821..751bc29c 100644
--- a/src/visualizer/OpenCvVisualizer3D.cpp
+++ b/src/visualizer/OpenCvVisualizer3D.cpp
@@ -613,11 +613,13 @@ void OpenCvVisualizer3D::visualizeMesh3D(const cv::Mat& map_points_3d,
     color_mesh = true;
   }
 
+  bool tcoords_mesh = false;
   if (tcoords.rows != 0) {
     CHECK_EQ(map_points_3d.rows, tcoords.rows)
         << "Map points and tcoords should have same number of rows. One"
            "tcoord per map point.";
     CHECK(!texture.empty());
+    tcoords_mesh = true;
   }
 
   // No points/mesh to visualize.
@@ -629,7 +631,7 @@ void OpenCvVisualizer3D::visualizeMesh3D(const cv::Mat& map_points_3d,
   cv_mesh.cloud = map_points_3d.t();
   cv_mesh.polygons = polygons_mesh;
   cv_mesh.colors = color_mesh ? colors.t() : cv::Mat();
-  cv_mesh.tcoords = tcoords.t();
+  cv_mesh.tcoords = tcoords_mesh ? tcoords.t() : cv::Mat();
   cv_mesh.texture = texture;
 
   // Plot mesh.

[marcusabate]

This or using Opencv 3.4.2 (on 20.04) works on my side.