enable_loop to True causes error after awhile, false does not result in error

[Melvin-leo]

d2vins_node: /root/swarm_ws/src/D2SLAM/d2frontend/src/loop_detector.cpp:561: bool D2FrontEnd::LoopDetector::computeCorrespondFeatures(const D2Common::VisualImageDesc&, const D2Common::VisualImageDesc&, std::vector<Eigen::Matrix<double, 3, 1> >&, std::vector<int>&, std::vector<Eigen::Matrix<double, 3, 1> >&, std::vector<int>&, std::vector<int>&): Assertion `img_desc_a.spLandmarkNum() * params->superpoint_dims == img_desc_a.landmark_descriptor.size() && "Desciptor size of new img desc must equal to to landmarks*256!!!"' failed.
Stack trace (most recent call last) in thread 6789:
#12   Object "", at 0xffffffffffffffff, in 
#11   Source "../sysdeps/unix/sysv/linux/x86_64/clone.S", line 95, in __clone [0x7fadfb324352]
#10   Source "/build/glibc-LcI20x/glibc-2.31/nptl/pthread_create.c", line 477, in start_thread [0x7fadfdabb608]
#9    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28", at 0x7fadfb639df3, in 
[08:50:20][info][d2featuretracker.cpp,trackLocalFrames,L136] frame_id: 1000628 is_kf false, landmark_num: 203/252, mean_para 2.20%, time_cost: 5.3ms 
#8    Source "/root/swarm_ws/src/D2SLAM/d2frontend/src/d2frontend.cpp", line 172, in loopDetectionThread [0x7fadfd7449ec]
        169:       if (loop_queue.size() > 10) {
        170:         SPDLOG_WARN("Loop queue size is {}", loop_queue.size());
        171:       }
      > 172:       loop_detector->processImageArray(vframearry);
        173:     }
        174:     loop_detection_thread_rate_ptr_->sleep();
        175:   }
#7    Source "/root/swarm_ws/src/D2SLAM/d2frontend/src/loop_detector.cpp", line 185, in processImageArray [0x7fadfd7012eb]
        182:                _old_fisheye_img.frame_id);
        183:         swarm_msgs::LoopEdge ret;
        184:         if (_old_fisheye_img.drone_id == self_id) {
      > 185:           success = computeLoop(_old_fisheye_img, image_array, camera_index_old,
        186:                                 camera_index, ret);
        187:         } else if (image_array.drone_id == self_id) {
        188:           success = computeLoop(image_array, _old_fisheye_img, camera_index,
#6    Source "/root/swarm_ws/src/D2SLAM/d2frontend/src/loop_detector.cpp", line 655, in computeLoop [0x7fadfd6fe3bc]
        652:   std::vector<int> camera_indices;
        653:   std::vector<std::pair<int, int>> index2dirindex_a, index2dirindex_b;
        654: 
      > 655:   success = computeCorrespondFeaturesOnImageArray(
        656:       frame_array_a, frame_array_b, main_dir_a, main_dir_b, lm_pos_a,
        657:       lm_norm_3d_b, camera_indices, index2dirindex_a, index2dirindex_b);
[08:50:20][info][d2featuretracker.cpp,trackLocalFrames,L136] frame_id: 1000629 is_kf true, landmark_num: 203/251, mean_para 3.43%, time_cost: 5.6ms 
#5    Source "/root/swarm_ws/src/D2SLAM/d2frontend/src/loop_detector.cpp", line 499, in computeCorrespondFeaturesOnImageArray [0x7fadfd6fb7b1]
        497:     if (dir_a < frame_array_a.images.size() &&
        498:         dir_b < frame_array_b.images.size() && dir_a >= 0 && dir_b >= 0) {
      > 499:       bool succ = computeCorrespondFeatures(
        500:           frame_array_a.images[dir_a], frame_array_b.images[dir_b], _lm_pos_a,
        501:           _idx_a, _lm_norm_3d_b, _idx_b, _camera_indices);
        502:       SPDLOG_INFO(
#4    Source "/root/swarm_ws/src/D2SLAM/d2frontend/src/loop_detector.cpp", line 561, in computeCorrespondFeatures [0x7fadfd6f7e4a]
        558:     _matches =
        559:         superglue->inference(kpts_a, kpts_b, desc0, desc1, scores0, scores1);
        560:   } else {
      > 561:     assert(img_desc_a.spLandmarkNum() * params->superpoint_dims ==
        562:                img_desc_a.landmark_descriptor.size() &&
        563:            "Desciptor size of new img desc must equal to to landmarks*256!!!");
        564:     assert(img_desc_b.spLandmarkNum() * params->superpoint_dims ==
#3    Source "/build/glibc-LcI20x/glibc-2.31/assert/assert.c", line 101, in __assert_fail [0x7fadfb238fd5]
#2    Source "/build/glibc-LcI20x/glibc-2.31/assert/assert.c", line 92, in __assert_fail_base [0x7fadfb227728]
#1    Source "/build/glibc-LcI20x/glibc-2.31/stdlib/abort.c", line 79, in abort [0x7fadfb227858]
#0    Source "../sysdeps/unix/sysv/linux/raise.c", line 51, in raise [0x7fadfb24800b]
Aborted (Signal sent by tkill() 6754 0)
[d2vins-1] process has died [pid 6754, exit code -6, cmd /root/swarm_ws/devel/lib/d2vins/d2vins_node __name:=d2vins __log:=/root/.ros/log/7facdba2-9cc7-11ef-9ba3-28d0ea3d2525/d2vins-1.log].
log file: /root/.ros/log/7facdba2-9cc7-11ef-9ba3-28d0ea3d2525/d2vins-1*.log

Really thankful of your amazing work, but there is this error that about the mismatch in dimensions for loop detection, maybe you have a way to resolve the desc size

[deepConnectionism]

I ran into the same question. The error is caused by enable_loop:=true.

I have tried the main branch and other branches have encountered this problem, may I ask the owner to solve it？

[qjy0427]

Resizing landmark_descriptor to spLandmarkNum() * params->superpoint_dims seems feasible.

[YQyqyyq]

Resizing landmark_descriptor to spLandmarkNum() * params->superpoint_dims seems feasible.

I have encountered the same problem. Even after resizing, it still reports an error. Could you provide more information to help me?

[YYWumm]

Resizing landmark_descriptor to spLandmarkNum() * params->superpoint_dims seems feasible.

I have encountered the same problem. Even after resizing, it still reports an error. Could you provide more information to help me?

I also encountered this problem on the pr_fix_main branch. It seems that the error is caused by a mismatch between the number of landmarks and landmark_descriptor.

I printed the values from the assert statement:

• img_desc_a.spLandmarkNum() * params->superpoint_dims
• img_desc_a.landmark_descriptor.size()

I found that the latter is always smaller than the former, which suggests that in some cases, landmarks were not updated in sync with landmark_descriptor.

I found that in d2featuretracker.cpp, within the stereo optical flow function trackLK(left_frame, right_frame, type, use_lk_for_left_right_track);, that only updates landmarks :

right_frame.landmarks.emplace_back(lm);

but does not update landmark_descriptor.

Following the update logic from the single cam optical flow function report.compose(trackLK(frame)); (in the same file), simply synchronizing the update of landmark_descriptor resolves this assertion error.

After making this modification, I have not encountered further problems. I hope this helps you fix the issue.