Frozen when setting up contraints matrix with tracker.optimaze()?

[forestrywhale]

I'm using btrack in Jupyter notebook, it can successfully find tracks, but the tracker.optimize() seems always frozen forever. Here is the out put in the screen:

[INFO][2022/02/19 05:48:26 PM]  - Found 8785 tracks in 577 frames (in 0.0s)
[INFO][2022/02/19 05:48:26 PM]  - Inserted 7225 dummy objects to fill tracking gaps
[INFO][2022/02/19 05:48:26 PM] Loading hypothesis model: cell_hypothesis
[INFO][2022/02/19 05:48:26 PM] Calculating hypotheses (relax: True)...
[INFO][2022/02/19 05:48:26 PM] Setting up constraints matrix for global optimisation...

After the above step, it will keep frozen forever. Even killing the kernel won't work. Is there any way I can solve this? Thanks a ton!

[forestrywhale]

So my solution is that somehow this is too many hypotheses to handle at one time. Thus I can split the .csv file as input, optimize and save the trace for different time intervals, and then bridge the traces. Technically this might not be a "global" optimize but probably works.

[quantumjot]

Indeed, this happens when you're generating too many hypotheses per track. It becomes difficult to opimize. Usually, this is because the configuration is not set up properly for your dataset. How many objects per frame / tracks are you expecting? ~9000 seems like a lot for most use cases.

[quantumjot]

Also, you can set a timeout for the optimizer by passing in some options, e.g.:

tracker.optimize(options={'tm_lim': int(6e4)})

This will stop the python script from freezing for a long time (max 60 seconds here).

[forestrywhale]

Thank you so much for your suggestion! This timeout function really helps! My images are drifting, so after I fixed that, I can get many traces up to 10 hours, and lower the number of trace to ~2500. The hypothesis tests were either unfinished or didn't fit my data so generations are not detected though.

[FrickTobias]

@forestrywhale Could I ask with what code changes you fixed it?

I am running in to similar issues where I am tracking ~5,000 cells at the time and it is currently being detected as ~38,000.

[INFO][2022/04/20 03:21:52 PM]  - Found 38061 tracks in 9 frames (in 0.0s)
btrack.core - INFO:  - Found 38061 tracks in 9 frames (in 0.0s)
[INFO][2022/04/20 03:21:52 PM]  - Inserted 206 dummy objects to fill tracking gaps
btrack.core - INFO:  - Inserted 206 dummy objects to fill tracking gaps
[INFO][2022/04/20 03:21:52 PM] Loading hypothesis model: cell_hypothesis
btrack.core - INFO: Loading hypothesis model: cell_hypothesis
[INFO][2022/04/20 03:21:52 PM] Calculating hypotheses (relax: True)...
btrack.core - INFO: Calculating hypotheses (relax: True)...
worker_process - INFO: Setting up constraints matrix for global optimisation...