Add multi-animal support to DLCLive

[jeylau]

• [ ] Check DLCRNet model freezing
• [ ] Add dependency on DLC??

MultiAnimalDLCLive can be instantiated e.g. as:

from dlclive.dlclive import MultiAnimalDLCLive

live = MultiAnimalDLCLive(
    'path_to_snapshot',
    n_animals=3,
    n_multibodyparts=12,
    track_method="box",
    sim_threshold=0.3,
)
live.init_inference(allow_growth=True)
animals, unique = live.get_pose(frame)

Fixes #57

[sneakers-the-rat]

Hell ya. thanks for handling this. I can't push to your fork, so I made a few initial hopefully uncontroversial amendments in madlc branch. We can either use that as the PR source or else you can just merge that into yours.

• Tests fail because scipy and deeplabcut are now imported but not added to package requirements in pyproject.toml, added (deeplabcut^2.2.0 and scipy^1.8.0 for python>=3.8 and >=1.7.0,<1.8.0 for python>=3.7.1,<3.8)
• Version not incremented per semver, incremented to 1.1.0 because of addition of new backwards-compatible feature
• Added CHANGELOG.md to describe changes to library since we don't have docs to speak of. Would you mind filling that out/adding some usage notes to the README?
• Added CONTRIBUTING.md to formalize the update process a tiny tiny bit: just stuff like increment version, write docs, tests, etc.
• Can we get a test for this? it can be as barebones as the existing one (just checking that it runs, rather than confirming the values of the run) imo. This will save us headaches down the line and prevent accidental regressions in any future changes. The test action just runs dlc-live-test (dlclive.check_install.check_install.main), so you could add it as an additional function there called within main(). Like the existing test, that would require
  • Downloading a pretrained model & project
  • downloading a test video
  • running inference on the test video

[thomasilmer]

@jeylau Are you still working on this? Otherwise I could attempt to make the requested changes.

[AlexEMG]

Also consider: https://forum.image.sc/t/dlc-live-multianimal-what-is-affinity-in-inferenceutil-py-empty-assemblies-bug-or-bad-model/69972

[jeylau]

Hey @thomasilmer, I'll update this PR soon, but please feel free to contribute too; you'd be very welcome!

[MMathisLab]

@jeylau let's finish this :)

[thomasilmer]

Hey @jeylau, I took a look at the requested changes and must admit that I quickly realized that I am not familiar enough with the internal workings of DeepLabCut to contribute anything meaningful at this moment.

My apologies.

[frankiechang123]

Hi @jeylau @MMathisLab !I'm currently trying to apply maDLC models on live videos. Just wondering what is the status of supporting maDLC models in DLCLive? I'm also willing to contribute to help the development!

[thomasilmer]

I have tried running this code locally and there were some changes I had to make to prevent exceptions from happening when indexes went out of bounds. What is the preferred method for contributing these changes @jeylau ?

Additionally, I noticed that track_method='ellipse' seems to give more consistent poses than track_method='box'. Is there any reason for this or is this heavily dependent on the type of data you process?

Finally, there is some strange behavior when analysing videos where animals enter the frame from one side and exit on the other side (have not tested on other footage). The animals are lost after some time and we need to run 'init_inference' again to get better results again.

[MMathisLab]

Very happy to accept a PR - so please just edit in your forked copy and open a pull request to us and tag us for review, thanks!!

Cc @n-poulsen

[itisomar]

Hi guys @gkane26 @jeylau @MMathisLab @sneakers-the-rat , first things first, a huge thanks and shout out to the DLC team and the people who are contributing to this wonderful project in any way whatsoever.

As I implemented the MultiAnimalDLCLive class, I encountered several issues that are known to you guys and the great DLC community.

I'll list the issues which I have encountered below:

1 - init_inference() : When you call init_inference(allow_growth=True) before get_pose(frame), the pose estimation seems to work well. However, in the long run, the pose estimation drops drastically in frames. In some cases, it fails to detect any body parts. One workaround is to call init_inference(allow_growth=True) whenever that happens, which asks for a lot of memory, causing the frame rate to drop to zero.” As some have mentioned in the following forum: https://forum.image.sc/t/live-inference-for-multi-animal-deeplabcut/73509.diff

2 - max_age : It needs to be set as high as ‘100’ for my project, in order for MultiAnimalDLCLive to be able to detect body parts. Otherwise, MultiAnimalDLCLive is unable to detect body parts and the detection rate drops drastically, near to zero detections.

3 - track_method : The ‘Box’ tracking method detects far fewer body parts in frames compared to the ‘Ellipse’ tracking method