[wip] pyramidal optical flow lk

[edgarriba]

Changes

part of #1381

• still not working, for some reason it does not converge

Type of change

• [ ] 📚 Documentation Update
• [x] 🧪 Tests Cases
• [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
• [x] 🔬 New feature (non-breaking change which adds functionality)
• [ ] 🚨 Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] 📝 This change requires a documentation update

Checklist

• [ ] My code follows the style guidelines of this project
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] My changes generate no new warnings
• [ ] Did you update CHANGELOG in case of a major change?

[Parskatt]

@edgarriba The hyperparams don't seem optimized to me, do you have a reference implementation you are referring to? For example, using these gradients: https://github.com/kornia/kornia/blob/251123a1ba29bdf2e32bfcf12176d8fa68ac9454/kornia/tracking/optical_flow_lk.py#L35 and having a window size of 3 does not seem good for regions with little structure.

Opencv has as default window size 21: https://docs.opencv.org/4.9.0/dc/d6b/group__video__track.html#ga473e4b886d0bcc6b65831eb88ed93323

and I think they use the schar derivative (although the code is almost unreadable to me): https://github.com/opencv/opencv/blob/76548e29bd62f299386cb919824b34ce4baf73cb/modules/video/src/lkpyramid.cpp#L804

I think "Optimal Filters for Extended Optical Flow" by Scharr is probably a good reference, although in opencv I guess probably its just these:

[edgarriba]

@Parskatt for the kernel i was following the paper summary description section where the author defines the derivatives with a central difference kernel (-1 0 1) -- and i just simplified to (-1 1).

Even though they explicitly say in practice is used Sharr operator. I Haven't tried (nor windinows size 21). Maybe we need to generate a better synthetic example and try again with that Kernel.

Are you interested in helping with this ? I'm recently focused on other tasks (which maybe i can try to port this to Rust when we have something working).

[Parskatt]

Ive TAd on this for a long time, so actually a bit tired of it. Might do it, but not prio.

[johnnv1]

as part of https://github.com/kornia/kornia/discussions/2719 plan, we have deleted the master in favor of main, if anyone has an interest in keep/restart working on this, let me know to help you.

[edgarriba]

i'd like to keep this one on the radar