[bug] Meshroom crashes during StructureFromMotion step without any error report

[tueboesen]

Describe the bug I am running a rather large dataset with ~18000 images, trying to capture the structurefrommotion (StM). The StM ran for 21 hours before crashing, and ended up using 75% of my memory, but did not appear to run out of memory.

The last lines of t he StM does not appear to give any information about what happened:

compressed_row_sparse_matrix.cc:174 # of rows: 354 # of columns: 354 max_num_nonzeros: 16380. Allocating 197980
[03:20:23.516225][info] Bundle Adjustment Statistics:
	- local strategy enabled: yes
	- graph-distances distribution:
	    - not connected: 0 cameras
	    - D = 0: 51 cameras
	    - D = 1: 0 cameras
	    - D > 1: 11209 cameras
	- adjustment duration: 0.446275 s
	- poses:
	    - # refined:  59
	    - # constant: 949
	    - # ignored:  10266
	- landmarks:
	    - # refined:  210556
	    - # constant: 0
	    - # ignored:  3186739
	- intrinsics:
	    - # refined:  0
	    - # constant: 1
	    - # ignored:  0
	- # residual blocks: 421112
	- # successful iterations: 5
	- # unsuccessful iterations: 0
	- initial RMSE: 0.499582
	- final   RMSE: 0.499519
[03:20:27.172238][info] Remove outliers: 
	- # outliers residual error: 0
	- # outliers angular error: 0
[03:20:29.303951][info] Bundle adjustment iteration: 1 took 7130 msec.
[03:20:29.304001][info] Bundle adjustment with 2 iterations took 21790 msec.
[03:20:50.622721][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5997
[03:20:51.143719][info] [11275/17277] Robust Resection of view: 1328376007
[03:21:13.369745][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5996
[03:21:13.897148][info] [11276/17277] Robust Resection of view: 1566355933
[03:21:34.864240][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5995
[03:21:35.398225][info] [11277/17277] Robust Resection of view: 194056304
[03:21:58.049786][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5994
[03:21:58.610283][info] [11278/17277] Robust Resection of view: 340051139
[03:21:58.620445][info] Resection of view 340051139 failed.
[03:22:21.101754][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5993
[03:22:21.641996][info] [11278/17277] Robust Resection of view: 309139985
[03:22:44.645672][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5992
[03:22:45.188945][info] [11279/17277] Robust Resection of view: 760959344
[03:23:08.147469][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5991
[03:23:08.681348][info] [11280/17277] Robust Resection of view: 80001495
[03:23:31.641718][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5990
[03:23:32.173508][info] [11281/17277] Robust Resection of view: 318989860
[03:23:54.984286][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5989
[03:23:55.506026][info] [11282/17277] Robust Resection of view: 955339205
[03:23:55.516017][info] Resection of view 955339205 failed.
[03:24:16.483091][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5988
[03:24:17.008669][info] [11282/17277] Robust Resection of view: 165761241
[03:24:39.696814][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5987
[03:24:40.229940][info] [11283/17277] Robust Resection of view: 296634524
[03:24:40.239671][info] Resection of view 296634524 failed.
[03:25:02.795729][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5986
[03:25:03.320517][info] [11283/17277] Robust Resection of view: 1056174934
[03:25:25.418625][info] Update Reconstruction:
	- resection id: 610
	- # images in the resection group: 1
	- # images remaining: 5985
[03:25:25.938920][info] [11284/17277] Robust Resection of view: 1518868145
[03:25:26.389550][info] Bundle adjustment start.
[03:25:30.096216][fatal] map::at

The only thing noticeable is that it appears to have failed in a bad way since it does not even manage to write the error to the logfile, but stops halfway through writing it.

My best bet is that it ran out of memory, since that is the only thing I can think of that might have given such a fatal error, but the statistics seems to indicate that meshroom was only using 75% memory at the time of the crash, and the OS does not use that much.

Desktop (please complete the following and other pertinent information): I am using Meshroom 2023.3.0 running on Ubuntu 22.04 Python version 3.10.12