elf icon indicating copy to clipboard operation
elf copied to clipboard

Enable linking across missing segments with motile based tracking

Open constantinpape opened this issue 8 months ago • 1 comments

The default tracking with motile from https://github.com/constantinpape/elf/blob/master/elf/tracking/motile_tracking.py#L5, which just uses the overlap to construct edge costs, works well for many tracking problems. However, it is currently limited by the fact that it cannot cross gaps (i.e. can only link objects if they are present in each consecutive frame or slice). Such gaps often happen due to:

  • Occlusion
  • Objects going in and out of frame
  • Segmentation errors

We should extend the tracking functionality so that objects can be tracked with gaps. I can see two possible implementations for this.

1. Including edges that link objects across more than one frame in the tracking problem:

This could be done by extending compute_edges_from_overlap to include a parameter that determines the maximal gap to be closed (in terms of frames), and then computing the edges and overlap for the corresponding adjacent frames instead of just the nearest adjacent frame. This should also include an option for a penalty that grows with the linking distance, so that links with smaller gaps are preferred.

2. Adding new functionality that links tracklets

This would add a second layer problem on top of the first tracking problem, where the nodes correspond to tracklets (the result from the first tracking run), and edges can link the tracklets and can allow linking across gaps. The advantage of this approach is that more expressive features can be computed for the edges between tracklets, e.g. based on how well the local trajectories align.

I think we should go with the first option for now:

  • it is much easier to implement, since we just need to update the function that computes the edges and can just re-use the rest of the current functionality.
  • for the simple overlap based features it should also yield better results, because the gap closing is solved together with the rest of the tracking problem rather than as a second separate step.

We should then also add a function that post-processes the gaps after tracking and inserts the interpolated segmentation masks.

constantinpape avatar Oct 30 '23 09:10 constantinpape