Practicality of Rolling Shutter Reconstruction

[psalvaggio]

trafficstars

Hello,

I have been talking for a while with the folks over at OpenDroneMap about how to add rolling shutter camera reconstruction to their workflow. Their user community is very interested in this use case, as most drone-based camera systems are rolling shutter. After digging into their codebase a bit, I found that they are leveraging OpenSFM for the photogrammetric workflow, so in addition to the work needed on their end, there would need to be support for a rolling shutter camera model in OpenSFM for the work to be practical. So, I wanted to open an issue here to see what the scope of the effort would be in order to add such support.

In my experience, adding this type of camera model is a multi-part effort. Normally it would require the following:

1. The actual camera model. For a drone application where there is a stabilizing gimbal, rolling shutter distortion primarily comes from aircraft motion. As such, I think a linear model would likely be sufficient.
2. Since the camera model has more DoF, more feature points are required and those feature points need to span the FOV of the camera in order to stably fit the motion parameters. In order to get more point, I have found it either requires more aggressive parameters on an existing feature extractor or a new extractor.
3. If epipolar constraints are used in confining feature matching, then those tolerances either need widened or a new constraint to account for an initial estimate of the rolling shutter distortion needs developed. If using global matching, nothing needs to change.
4. Since from 2 there are a lot more matches in the system, tie points need filtered down to eliminate mismatches and ensure a good spread of tie points in the space of each image. I am not sure if OpenSFM support blocking, but if so block sizes tend to have to be smaller for rolling shutter problems.
5. The bundle adjustment needs to be adjusted to support the new camera model.

Have you guys scoped this effort out at all previously? If not, could we start a discussion on this front? I would be interested in helping in its development.

Thanks, Phil

[Saijin-Naib]

Hate to be a pain, but just following up to see if anyone had any further clarity on this.

We've begun working something out on our end:
https://github.com/OpenDroneMap/ODM/pull/1467

But it is still early-stage and we'd appreciate any further clarity, expertise, and advice folks might have for us (especially since ideally, it makes its way upstream!). It goes without saying, but I'll say it anyway: Obviously, any and all collaboration is more than welcome as well!

[pierotofy]

As an update, I've implemented a rs_correct command in https://github.com/OpenDroneMap/OpenSfM/blob/286/opensfm/actions/rs_correct.py which modifies/trims the features from a reconstruction to correct for rolling shutter using the method outlined in "A two-step approach for the correction of rolling shutter distortion in UAV photogrammetry" from https://www.sciencedirect.com/science/article/abs/pii/S0924271619302849. Then the matching/create_tracks/reconstruction steps can be re-run using the corrected features.

I'd be happy to open a PR to bring this upstream, if there's interest. Let me know.