crash during run_custm.py

[bblumberg]

I am using the docker version of bundlesdf running on an x86 machine with a 4070 ti.

• To get things running I had to uninstall and reinstall scipy, and pip install yacs
• Once those little things were fixed, I ran run_custom.py on the milk example. It runs for 15 minutes or so, generating output which appears normal before spitting out the error below. Any ideas?

Thanks,

bruce

-[bundlesdf.py] frame 1668813071116728056 pose update after [[-0.709 -0.681 -0.182 0.105] [-0.695 0.632 0.342 -0.081] [-0.118 0.369 -0.922 0.437] [ 0. 0. 0. 1. ]] [2025-01-14 06:53:16.050] [warning] [Bundler.cpp:435] total keyframes=186, want to select 10 [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:516] ids#=186, max_BA_frames-frames.size()=9 [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:525] frames#=10 [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813056725965400 visible=0.813003 [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813056725965400) into pairs [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813056725965400 visible=0.789516 [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813056725965400) into pairs [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813057005518533 visible=0.811209 [2025-01-14 06:53:16.090] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813057005518533) into pairs [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813057005518533 visible=0.789139 [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813057005518533) into pairs [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813057204255187 visible=0.778267 [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813057204255187) into pairs [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813057204255187 visible=0.753044 [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813057204255187) into pairs [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813070042892544 visible=0.856001 [2025-01-14 06:53:16.091] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813070042892544) into pairs [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813070042892544 visible=0.834662 [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813070042892544) into pairs [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813070174291989 visible=0.892966 [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813070174291989) into pairs [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813070174291989 visible=0.866124 [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813070174291989) into pairs [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813070341765204 visible=0.907371 [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813070341765204) into pairs [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813070341765204 visible=0.883041 [2025-01-14 06:53:16.092] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813070341765204) into pairs [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813070574800195 visible=0.913079 [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813070574800195) into pairs [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813070574800195 visible=0.894839 [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813070574800195) into pairs [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:793] frame 1668813071086837259 and 1668813070957531326 visible=0.937269 [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:802] add frame (1668813071086837259, 1668813070957531326) into pairs [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:793] frame 1668813071116728056 and 1668813070957531326 visible=0.919298 [2025-01-14 06:53:16.093] [warning] [Bundler.cpp:802] add frame (1668813071116728056, 1668813070957531326) into pairs [bundlesdf.py] frame_pairs: 16 [2025-01-14 06:53:16.093] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813056725965400) [2025-01-14 06:53:16.094] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813057005518533) [2025-01-14 06:53:16.095] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813057204255187) [2025-01-14 06:53:16.096] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813070042892544) [2025-01-14 06:53:16.097] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813070174291989) [2025-01-14 06:53:16.098] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813070341765204) [2025-01-14 06:53:16.099] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813070574800195) [2025-01-14 06:53:16.100] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (1668813071086837259, 1668813070957531326) [loftr_wrapper.py] image0: torch.Size([8, 1, 400, 400]) Traceback (most recent call last): File "run_custom.py", line 222, in run_one_video(video_dir=args.video_dir, out_folder=args.out_folder, use_segmenter=args.use_segmenter, use_gui=args.use_gui) File "run_custom.py", line 102, in run_one_video tracker.run(color, depth, K, id_str, mask=mask, occ_mask=None, pose_in_model=pose_in_model) File "/home/easybot/workspaces/BundleSDF/bundlesdf.py", line 543, in run self.process_new_frame(frame) File "/home/easybot/workspaces/BundleSDF/bundlesdf.py", line 494, in process_new_frame self.find_corres(pairs) File "/home/easybot/workspaces/BundleSDF/bundlesdf.py", line 362, in find_corres corres = self.loftr.predict(rgbAs=imgs[::2], rgbBs=imgs[1::2]) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/home/easybot/workspaces/BundleSDF/loftr_wrapper.py", line 50, in predict self.matcher(tmp) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/easybot/workspaces/BundleSDF/BundleTrack/LoFTR/src/loftr/loftr.py", line 49, in forward feats_c, feats_f = self.backbone(torch.cat([data['image0'], data['image1']], dim=0)) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/easybot/workspaces/BundleSDF/BundleTrack/LoFTR/src/loftr/backbone/resnet_fpn.py", line 116, in forward x1_out = self.layer1_outconv2(x1_out+x2_out_2x) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward input = module(input) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 447, in forward return self._conv_forward(input, self.weight, self.bias) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 443, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

[pithreeone]

I'm also facing this issue. Has anyone found a solution?

[umurotti]

Same here

[ArghyaChatterjee]

Same here, did anyone solved the problem ? It happened to me after processing 700 images. How to solve this one ??