Improve point trajectory estimation by aggregating across sources

[sfmig]

The simplest case may be centroid tracking, for example obtained with idtracker and SLEAP independently.

The question is: can we combine these two (or more) sources of information to produce a more accurate trajectory for the centroid?

This may be more relevant for a multi-animal case in which there are id swaps, but the two methods to get the data may fail in different cases.

Some options could be: to use the most reliable one to correct the other (for a given definition of reliable), take some kind of consensus, consider sensor fusion approaches like Kalman filter, or taking the rolling median filter using both sources of data. Another approach could be taking the mean of both sources (so somewhat related to this #271).

Some sample data that we could use for this is the EPM:

• DLC_single-mouse_EPM.predictions.csv (or the equiv .h5 file) as the "more reliable" source, and
• SLEAP_single-mouse_EPM.analysis.h5 (or the equivalent .slp file), which is generated using a model trained on less data (should be less reliable).
• They differ in the number of keypoints so we could using the intersection set, or the centroid for this analysis.

[roaldarbol]

For reference, trex can output "tracklets" or each individual: Small egocentric videos of each individual (centered on their centroid) which can then be tracked using SLEAP or DLC. Haven't tried it myself though.

[sfmig]

thanks @roaldarbol for pointing this out, idtracker can do this too (see the tip box here)

[roaldarbol]

Ah, so cool, didn't know they did it too!!!

[parikshit14]

Hi @sfmig, is this issue still relevant? I would be happy to take it up.

[sfmig]

Hi @parikshit14, yes we are still interested in exploring how to best support this.

You can find the datasets mentioned above in the movement test data repo. You can see an example on how to access them here. Other datasets that may be useful are:

• SLEAP_two-mice_social-interaction.analysis.h5 and SLEAP_two-mice_social-interaction.predictions.slp (I am not sure if there is a more reliable one)
• SLEAP_three-mice_Aeon_mixed-labels.analysis.h5 as the "less reliable" and SLEAP_three-mice_Aeon_proofread.analysis.h5 as the more reliable.

Let me know if you need any help!

[niksirbi]

SLEAP_two-mice_social-interaction.analysis.h5 and SLEAP_two-mice_social-interaction.predictions.slp (I am not sure if there is a more reliable one)

I think these two files contain the exact same data in different formats, so probably not very helpful for trying out aggregation approaches.

SLEAP_three-mice_Aeon_mixed-labels.analysis.h5 as the "less reliable" and SLEAP_three-mice_Aeon_proofread.analysis.h5 as the more reliable.

This one should work for that purpose though.