Solve Lens Distortion

[david-cattermole]

Description

mmSolver can solve transform attributes with normal CG rectilinear lenses, but cannot currently model Lens distortion. We should provide support for solving Lens Distortion.

To solve lens distortion we must be able to model the non-uniform 2D screen-space offsets in both 2D Markers and the displayed image plane.

2D Markers will need to be connected to lens distortion nodes, directly or indirectly. Marker Maya nodes will require a custom 3D transform node to be written in Maya API for us to visualize the correct 2D transform.

Creating a new Maya imagePlane that is able to undistort in real-time and rival the features of Maya's Image Plane is considerable effort, and will be avoided initially. Instead we should create a deformer node (MPxDeformer) that is able to deform a Polygon Plane. The Polygon Plane should then have a File Texture shader assigned and will allow the distorted image to be undistorted, in real-time.

Supporting all models of Lens distortion found in common 3D tracking software is out of the reach of this project. Instead we will focus on using the most common, and most advanced system; Science d Visions (3DEqualizer) Lens Distortion Plug-in Kit, also known as the LDPK. Adding LDPK to the CMake build system will be needed. It will also be helpful to compile the Python bindings for LDPK, in case we wish to use the LDPK in the Maya Python tools.

Usage

1. Create Camera
2. Add Maya Image Plane
3. Add Marker to Camera
4. Select Maya Image Plane
5. Run tool; Image Plane will convert into a polygon plane with (animated) File Texture attached. Lens Distortion set up, ready for solving.
6. Select lens distortion attributes in Maya channel box, add to Solver UI Attributes
7. Solve lens distortion values.

Notes

This GitHub issue will ignore the need to export or use the solved lens distortion values and instead leaves that problem for a Maya Python tool-based approach.

[david-cattermole]

I have come back to the solver_issue106 branch and I have noticed there are a few bugs/problems still to be resolved before release.

• Image Planes do not automatically connect to the camera (focal length, film back, etc).
• Lens distortion nodes are hard to find in the Maya UI.
  • Perhaps we need a "Show Lens Distortion in AE" tool, to automatically find and open the Lens Distortion node(s) in the Maya Attribute Editor.
• Solving of lens distortion parameters may not be handled by the solver very well.
• It is not clear how to use the image plane and lens distortion nodes.
  • For example what order must they be created in?
• The "Camera Tools" shelf button needs to have the "CTools" label, not "Cam".
• Centering on 3D transform nodes takes a long time to connect, this could be caused by Solver UI updates while the script is running.
  • This problem is possibly unrelated to lens distortion.

[david-cattermole]

Centering on 3D transform nodes takes a long time to connect, this could be caused by Solver UI updates while the script is running.

This problem is possibly unrelated to lens distortion.

Confirmed, this problem only happens when the Solver UI is open. I've created a new issue for it; #145.

[david-cattermole]

The future direction of mmSolver has been to support real-time lens distortion, for v0.4.x.

To support real-time lens distortion, we will be adding a new image plane (#173) node.

Also, I would like to have the following goals for the lens distortion and image plane combination:

• Use 3DEqualizer LDPK lens distortion models.
• Solve static values and per-frame lens distortion values inside mmSolver.
• Adjust lens distortion parameters and get real-time visual updates in Maya viewport 2.0.
• Export a distorted image sequence.
• Export a ST-Map EXR file(s), even for layered lens distortion.
• Import Lens Distortion values from 3DEqualizer and Nuke .nk files
• Import 3DEqualizer Lens Matrixes into Maya.
• Possible support for SynthEyes lens distortion models and SynthEyes imports.

I believe most or all of this support is needed before the tool can be useful in production.

[david-cattermole]

I almost forgot about this issue/branch.

The branch solver_issue106 contains a half implemented lens undistortion toolset that will be very helpful when integrating with OpenCompGraphMaya as part of #173.

[david-cattermole]

Lens distortion solving has been added in 370b5dc but is disabled by default, and is controlled with the #define MMSOLVER_LENS_DISTORTION in the file src/mmSolver/adjust/adjust_defines.h on the develop branch.

The current implementation increases solve time in the Maya DAG from ~110 seconds to ~200 seconds. The same implementation increases solve time in the MM Scene Graph from ~8 seconds to ~130 seconds.

The MM Scene Graph implementation also causes instability, with the same scene crashing sometimes (and not other times), as well as mmSolver producing a vastly different solve from the Maya DAG.

This slow-down is significant, and slows-down all solves, regardless of whether lens distortion is being solved or not.

This implementation is currently unacceptable, but a good start to be refined.

[david-cattermole]

On a more positive note, when solving lens distortion, some test scenes are solving with a slightly lower deviation and the lens distortion is not crazy - which is great!

[david-cattermole]

Closing. This has been released in v0.4.0.

We now have a mmLensModel3de node that can be used and connected to cameras. Markers and MM Image Planes can be adjusted using the Lens distortion attributes in real time, for fast user-feedback.