Wrong answer(or wrongly use) for cv2.optflow.calcOpticalFlowSparseRLOF in opencv-python

[tianty]

System information (version)

• OpenCV => :4.4.0.42
• Operating System / Platform => :window 10 64 bit
• Compiler => :gcc

Detailed description

I'm trying to use cv2.optflow.calcOpticalFlowSparseRLOF() to tracker the sparse point in frame series. But in the experiment, I found that if I just input one point to cv2.optflow.calcOpticalFlowSparseRLOF(), it can work well. However, if I input more than one points to the function, it can't calc the optflow and return same numbers of the input points. And the same input can work well in cv2.calcOpticalFlowPyrLK(). I have checked the type and the dimension of the input points numpy array, it is same to the result of cv2.goodFeaturesToTrack(). I don't kown what happend, thanks for the help!

Steps to reproduce

from cv2 import cv2
import numpy as np
import glob

img_path = '../20200622/L'

images = glob.glob(img_path+'/*.'+'png')
images.sort()

feature_params = dict(maxCorners=100,
                      qualityLevel=0.3,
                      minDistance=7)
p0 = cv2.goodFeaturesToTrack(cv2.cvtColor(cv2.imread(images[0]), cv2.COLOR_BGR2GRAY), mask=None, **feature_params)

pts0 = np.array([[1092,534],[920,404]]).reshape(-1,1,2)
pts1 = np.zeros(np.shape(pts0))
pts0 = np.float32(pts0)
pts1 = np.float32(pts1)

for i in range(1,len(images)):
    print('loop: {}'.format(i))
    pimg = cv2.imread(images[i-1])
    nimg = cv2.imread(images[i])
    pimg_g = pimg
    nimg_g = nimg

    pts1, status, err = cv2.optflow.calcOpticalFlowSparseRLOF(pimg_g, nimg_g, pts0, pts1)
    # lk_params = dict( winSize  = (15, 15), 
    #               maxLevel = 2, 
    #               criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))  
    # pts1, status, err = cv2.calcOpticalFlowPyrLK(pimg_g, nimg_g, pts0, None) #, **lk_params)

    pts0 = pts1

Issue submission checklist

• [ ] I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, answers.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version and the issue is still there

• [ ] There is reproducer code and related data files: videos, images, onnx, etc

[alalek]

Please try to capture input of failed case and provide minimal complete reproducer (including input images in lossless .png format - see imwrite())

I have checked the type and the dimension of the input points numpy array

Consider adding dump from cv.utils.dumpInputArray(value).

[tylernewnoise]

Unfortunatly I can somehow recreate this.

System information (version)

• Opencv 4.4.0 installed in a venv from opencv-contrib-python (I know that this is not official)
• Opencv 4.5.0 installed from arch linux packages
• reproduced in both versions
• Operating system: Arch Linux

Detailed description When tracking feature points with cv.optflow.calcOpticalFlowSparseRLOF() there is some undefined behaviour. This is is somewhat mysterious, but here's the deal:

import numpy as np
import cv2 as cv

pimg = cv.imread('/usr/share/opencv4/samples/data/basketball1.png', None)
nimg = cv.imread('/usr/share/opencv4/samples/data/basketball2.png', None)

p0 = np.array([[129, 122],[130, 111], [100, 100]], dtype=np.float32)

p1, st, err = cv.optflow.calcOpticalFlowSparseRLOF(pimg, nimg, p0, None)
# print(p1)
p1, st, err = cv.optflow.calcOpticalFlowSparseRLOF(pimg, nimg, p0, None)
print(p1)

Images are from the opencv-samples, the basketball1.png and basketball2.png.

• When I call cv.optflow.calcOpticalFlowSparseRLOF() twice for multiple feature points, I get reasonable results.
• When I call cv.optflow.calcOpticalFlowSparseRLOF() twice for multiple feature points but uncomment the print(p1) in between the calls, I get undefined points.
• When I call cv.optflow.calcOpticalFlowSparseRLOF() once for multiple feature points, I get undefined points.
• When I call cv.optflow.calcOpticalFlowSparseRLOF() once for one feature point, I get [[0. 0.]] as output.
• When I call cv.optflow.calcOpticalFlowSparseRLOF() twice for one feature point, I get reasonable results.

I have no idea why the call has to be twice and, even more mysterious to me, how the print-statement can somehow change the results. :thinking:

[Soongja]

@alalek Will this bug be fixed in the next release version?

[jyh2378]

In case opencv-python and opencv-contrib-python==4.6.0.66, I think this bug is still being found.

Reproducible example Sample images: Frame 1, Frame 2, text_mask

import numpy as np
import cv2

frame1 = cv2.imread("frame1.jpg")
frame2 = cv2.imread("frame2.jpg")
text_mask = cv2.imread("frame2.jpg", cv2.IMREAD_GRAYSCALE)

frame1_gray = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)

detector = cv2.ORB_create(100000, 1.2, 8, 31, 0, 0, cv2.ORB_HARRIS_SCORE, 31, 30)
keypoints = detector.detect(frame1_gray, text_mask)
prev_keypoints = np.float32([kp.pt for kp in keypoints]).reshape(-1, 1, 2)

next_points, status, err = cv2.optflow.calcOpticalFlowSparseRLOF(frame1, frame2, prev_keypoints, None)

M, inlier_mask = cv2.findHomography(prev_keypoints, next_points, cv2.USAC_MAGSAC)

In this example, I get unreasonable result next_points, and homography M can not be calculated. (On the other hand, same function in C++ and cv2.calcOpticalFlowPyrLK return reasonable results.)

SparseRLOF (Python)	SparseRLOF (C++)	SparsePyrLK

@tsenst I know you as an author of RLOF paper. If you don't mind, could you help this bug issue?

[gbartoli-zcs]

I am using OpenCV 4.10 on Conda and I have exactly the same problem: while SparsePyrLK works without any problem, running SparseRLOF on the same input gives messy coordinates as output also for valid matches (I worked on grayscale images and disabled some advanced features to reduce complexity):

rlof_params = cv.optflow.RLOFOpticalFlowParameter_create()
rlof_params.setUseIlluminationModel(False)
rlof_params.setUseGlobalMotionPrior(False)
rlof_params.setMaxIteration(50)
rlof_params.setSolverType(0)
rlof_params.setSupportRegionType(cv.optflow.SR_FIXED)

opt_flow = cv.optflow.SparseRLOFOpticalFlow_create(rlof_params)
next_pts, status, error = opt_flow.calc(prev_img_src, next_img_src, prev_pts, None)

>>> next_pts[:10]
array([[6.80125920e+24, 3.15936752e-41],
       [4.70285824e+08, 4.01808322e-41],
       [6.80125920e+24, 3.15936752e-41],
       [6.80125920e+24, 3.15936752e-41],
       [1.16940362e+30, 4.01794309e-41],
       [1.16954023e+30, 4.01794309e-41],
       [1.16992104e+30, 4.01794309e-41],
       [1.40129846e-45, 1.40129846e-45],
       [0.00000000e+00, 0.00000000e+00],
       [1.16936614e+30, 4.01794309e-41]], dtype=float32)
>>> status[:10]
array([[1],
       [0],
       [1],
       [1],
       [0],
       [1],
       [1],
       [1],
       [1],
       [0]], dtype=uint8)

It seems that this problem is present since 4 years, I hope the Python interface for RLOF will be fixed soon... :crossed_fingers:

PS: any news from @tsenst ?

[honnorat]

Still the same issue with opencv-contrib-python==4.11.0.86...