initUndistortRectifyMap huge lags

[mbaranovskyi]

Hello, I am trying to make camera control effect when user can rotate fisheye cameras image to look around but I am facing with huge performance problems when using such code

private void UndistortFisheye(Mat cameraMat, Quaternion rotation)
        {
            Profiler.BeginSample("UndistortFisheye");
            Mat map1 = new Mat();
            Mat map2 = new Mat();

            rotation.ToAngleAxis(out float angle, out Vector3 axis);

            var rot = axis * angle * Mathf.Deg2Rad;
            Mat rvec = new Mat(3, 1, CvType.CV_64F);

            rvec.put(0, 0, rot.x);
            rvec.put(1, 0, -rot.y);
            rvec.put(2, 0, rot.z);

            Mat R = new Mat(3, 3, CvType.CV_32FC1);

            Calib3d.Rodrigues(rvec, R);
            Profiler.BeginSample("fisheye_initUndistortRectifyMap");

            Calib3d.fisheye_initUndistortRectifyMap(
                                                    cameraMat,
                                                    m_distortionCoeffs,
                                                    R,
                                                    cameraMat,
                                                    m_size,
                                                    CvType.CV_16SC2,
                                                    map1,
                                                    map2
                                                   );

            Profiler.EndSample();

            Profiler.BeginSample("remap");

            Imgproc.remap(
                          m_inputFrame,
                          m_undistort,
                          map1,
                          map2,
                          Imgproc.INTER_LINEAR,
                          OpenCVForUnity.CoreModule.Core.BORDER_CONSTANT
                         );

            Profiler.EndSample();

            Profiler.BeginSample("matToTexture2D");

            Utils.matToTexture2D(m_undistort, m_outputTexture);
            Profiler.EndSample();

            Profiler.BeginSample("Apply");

            m_outputTexture.Apply();
            Profiler.EndSample();

            map1.Dispose();
            map2.Dispose();
            rvec.Dispose();
            R.Dispose();
            Profiler.EndSample();
        }

here is profiling data 0.5sec is pretty much

so I guess the main issue is image size my one has 2880x2880 but even decreasing it to 1024x1024 doesnt really help

Is it resolvable somehow? cause I have feeling that opencv for other frameworks works better

[mbaranovskyi]

I am curious if it is possible to make these calculation inside vertex or compute shader. I would try writing own compute shader but I can't find source code for fisheye_initUndistortRectifyMap and Imgproc.remap. Or do you have other ideas how to do it?

[matthewc84]

I was able to overcome this purely using undistort(), which claims to have the same effect. I am still struggling with the result, however, but the lag is gone.