co-tracker
co-tracker copied to clipboard
facebookresearch
possible applications
this result is impressive, does there any possible downstream applications examples which this tech can be used?
Thank you, @lucasjinreal! I believe models like CoTracker can be useful for many applications where motion estimation is needed. For example, video segmentation could be enhanced by combining point tracking with Segment Anything. I also think that CoTracker can improve 3D reconstruction of both static and dynamic objects, or even object tracking itself.
Furthermore, It can be directly applied in video editing to attach something, such as text or stickers, to a moving object.
@nikitakaraevv thanks, does the computation heavy (for example, able to run on mobile devices like snapchat visualeffects)? Also wanna ask, Does CoTracker able to get real-world 3d tracking results which could be used in 3d body estimation (3d body estimation suffering on tracking people or moving camera) ?
Hi @lucasjinreal! If you don't need to run it in real-time, I believe CoTracker can already work on mobile devices if you manage to convert it to the required format. However, a lot of work has to be done to optimize it for real-time applications on-device.
CoTracker currently estimates 2D tracks, but I think it should be possible to improve 3D body pose estimation even with such tracks. This looks like a research problem! :)
@nikitakaraevv Looks promising, so, is CoTracker able to run realtime on CPU? I think it can be more be usable if combined with appearing StableDiffusion's controlnet swap body from user's video.
Thanks, @lucasjinreal! CoTracker is currently unable to run in real-time on a CPU as it is still not fast enough for such a use-case. And thanks for your suggestion, I will try to add examples with CoTracker applications to this repo later.
Hi @nikitakaraevv,
Thank you for your amazing work. I am working on a project aimed at segmenting and reconstructing 3D objects from head-mounted AR device scanned data previously with mask-rcnn and tsdf integration. Recently I tried SAM and Co-tracker and found them to work very well and I'm considering integrating them into my work in the future.
You've previously noted that "CoTracker could improve the 3D reconstruction of both static and dynamic objects." Could you provide some specific insights into how CoTracker achieves this improvement? I'm particularly interested in understanding the underlying mechanics and the particular aspects where the enhancement is most noticeable. Your expertise on this subject would be greatly appreciated.
Thank you for your contribution to the field and for your time in addressing this inquiry.
Hi @DavidTu21, thank you!
Did you replace PIPs with CoTracker in this project? It would be interesting to see any numbers or results if you could share them!
We're currently working on 3D reconstruction using CoTracker, but we don't have any specific insights yet. This is just our current intuition. Stay tuned!
Hi @nikitakaraevv, thank you for your reply! I haven't replaced PIPs with CoTracker in that project, I was using a different tracking method with SAM previously which is called Track-Anything, and realised that I could also try the CoTracker for point-based tracking. I will give it a go and let you know if there are any interesting results :)
@nikitakaraevv Can co-tracker work real time on video stream from a webcam? Are there any examples for this?
Hi @lucasjinreal! If you don't need to run it in real-time, I believe CoTracker can already work on mobile devices if you manage to convert it to the required format. However, a lot of work has to be done to optimize it for real-time applications on-device.
CoTracker currently estimates 2D tracks, but I think it should be possible to improve 3D body pose estimation even with such tracks. This looks like a research problem! :)
This is a truly amazing project! My project involves the tracking of a moving camera (filming ice-skaters), and this algorithm truly outperforms anything that is out there in terms of point tracking accuracy and consistency. This is the first step in any camera tracking solution.
Unfortunately, the second step of camera tracking, what is commonly referred as "3D Solve" seems to be outside the scope of this research. Is anyone aware of a project that combines the outputs of co-tracker with a 3D camera solver ?
Hi @abhishekmonogram, yesterday, we released a new model that has an online mode, please see this issue.
Hi @GeorgeAdamon, thank you! Yes, we are working on that as well. You can check out our new paper https://vggsfm.github.io/
Thank you, @lucasjinreal! I believe models like CoTracker can be useful for many applications where motion estimation is needed. For example, video segmentation could be enhanced by combining point tracking with Segment Anything. I also think that CoTracker can improve 3D reconstruction of both static and dynamic objects, or even object tracking itself.
Furthermore, It can be directly applied in video editing to attach something, such as text or stickers, to a moving object.
Hi,
Thank you for your comprehensive paper; I really enjoyed reading it. I have a question regarding the application of version 2 (v2) in point tracking with the "Segment Anything" feature. I've read some of your comments about the performance drop of v2 in the previous code base. Could you please clarify if it's still feasible to use v2 for point tracking with "Segment Anything"?
Thank you!
Hi @AshkanTaghipour, I haven't tried it myself, but it should be similar to v1, only much more efficient. As a rule of thumb, the more points you are tracking at a time, the better the performance.
CoTracker is impressive! We are using it in Annolid for CoTracker3: Efficient Animal Tracking with Minimal Keypoint Labeling. NotebookLM deep dive: https://www.youtube.com/watch?v=xmoETgaGPQc .
hello, we've been using co tracker to track biological samples and keep them under the microscope field of view. I'm wondering now how we could use co-tracker to track cells inside the samples. Indeed, we have 3D movies and one idea would be to track in slices of different projections, but ideally, a real 3D implementation of the model would be awesome. From what I read this is among ideas you've had, so just checking in the status. Alternatively, what about re-training on our own 3D tracks? And when I say 3D, we really have volumetric data, we slice the biological sample in z.
