Unable to retrieve multiple predictions from maDLC trained network

[FabianPlum]

Hi there,

I'm working on a little Blender addon to make use of DLC-Live for multi-animal motion capture in foraging experiments where individuals freely enter and leave the scene. I'm passing cropped detections from a separate network (Yolo v4) to DLC-Live via OpenCV as the number of animals and joints are otherwise prohibitively large and not constant. The addon is taking shape and your DLC-Live implementation works great for the most part!

At the moment, however, I fail to retrieve multiple predictions at the same time.

Loading the graph and passing an image to initialise the network shows that there is only a single set of output predictions from sess with shape (num_keypoints , 3)

I have attached the pose_cfg.yaml file below, in case you can spot some issues in there that are not immediately obvious to me. Thanks a lot for your help!

Fabi

pose_cfg.yaml.txt

[AlexEMG]

Indeed, currently DLC-Live only supports single animals.

[FabianPlum]

Ah! Thanks for clarifying! I'll try to find a workaround with more elaborate cropping for now and report back if I find a nicer solution.

All the best Fabi

[nizheyi]

Indeed, currently DLC-Live only supports single animals.

May I ask for the progress of the mobilenet that support the multi-animal project? It would be of great help to my studies!

[thomasilmer]

Indeed, currently DLC-Live only supports single animals.

Are there any plans for multi animal support in DLC-Live?

[sneakers-the-rat]

I don't know the DLC internals well enough to answer this, but wonder if any of the other maintainers could comment on what this might take? we could give a sketch for a PR maybe?

[MMathisLab]

Hey there, it is currently a WIP in the lab (sorry, long list these days), but we just released 3D maDLC, which goes along this path too, see 2.2.1

[thomasilmer]

Hey there, it is currently a WIP in the lab (sorry, long list these days), but we just released 3D maDLC, which goes along this path too, see 2.2.1

Thanks for the answer.

Is there any rough estimate for when this will be done? And is there any possibility to offer financial compensation to speedup the implementation of this in DLC-live?

[niels-ilmer]

Would be great! really need this feature. @FabianPlum did you manage to make a good working workaround? How do you train the model to perform best?

[jeylau]

Guys, work is in progress here #72. It is functional (in my hands 😄), but will definitely require some polishing.