How to use DepthCleaner class with Python

[andreaceruti]

System information (version)

• OpenCV => 4.5.5 contrib
• Operating System / Platform => Ubuntu 18.04

Detailed description

I would like to use the class in order to clean my deoth image that has some black pixels, the problem is that since documentation is lacking I am unable to use it properly.

Steps to reproduce

import cv2
import numpy as np
image = cv2.imread("depth_image_path")
depthCleaner = cv2.rgbd.DepthCleaner_create(cv2.CV_16U, 7, cv2.rgbd.DepthCleaner_DEPTH_CLEANER_NIL )
output_image = np.copy(image)
depthCleaner.apply(image, output_image)

The problem is then that my output image looks exactly as the input image, so how would I use DepthCleaner class?

If from the snippet above I run

print(dir(depthCleaner))

I see all of its method but I can't understand their connection with the official .cpp implementation since I am not confident in using cpp. The avaliable methods are [ 'apply', 'clear', 'create', 'empty', 'getDefaultName', 'getDepth', 'getMethod', 'getWindowSize', 'initialize', 'read', 'save', 'setDepth', 'setMethod', 'setWindowSize', 'write']

Issue submission checklist

• [ ] I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution

• I updated to the latest OpenCV version and the issue is still there

• There is reproducer code and related data files: videos, images, onnx, etc

[berak]

as it is now, it only accepts CV_16U, CV_32F or CV_64F single channel depth images, but you are reading in a 3channel bgr , it falls through here, and just does nothing. so, asuming, your "depth_image_path" leads to a 16bit, single channel depth image, it would be:

image = cv2.imread("depth_image_path", cv2.IMREAD_UNCHANGED) # preserve 16bit
depthCleaner = cv2.rgbd.DepthCleaner_create(cv2.CV_16U, 7, cv2.rgbd.DepthCleaner_DEPTH_CLEANER_NIL )
output_image = depthCleaner.apply(image)

however, the docs / comments are indeed faulty and misleading (most of it seems copypasted from the rgbdNormals class):

https://github.com/opencv/opencv_contrib/blob/98f6a2e55453fdb109d18403ed546ac9d72161b4/modules/rgbd/include/opencv2/rgbd/depth.hpp#L210

DepthCleaner::DEPTH_CLEANER_NIL is the only implemented method here https://github.com/opencv/opencv_contrib/blob/98f6a2e55453fdb109d18403ed546ac9d72161b4/modules/rgbd/include/opencv2/rgbd/depth.hpp#L218-L219 no, it does not accept 3d point clouds, just single channel depth maps

https://github.com/opencv/opencv_contrib/blob/98f6a2e55453fdb109d18403ed546ac9d72161b4/modules/rgbd/src/depth_cleaner.cpp#L47 wrong paper

https://github.com/opencv/opencv_contrib/blob/98f6a2e55453fdb109d18403ed546ac9d72161b4/modules/rgbd/src/depth_cleaner.cpp#L78 there should be a default case, throwing an error for wrong input type (maybe even check type(), not depth() ?)

https://github.com/opencv/opencv_contrib/blob/98f6a2e55453fdb109d18403ed546ac9d72161b4/modules/rgbd/src/depth_cleaner.cpp#L55-L56 hmmm, unused

https://github.com/opencv/opencv_contrib/blob/98f6a2e55453fdb109d18403ed546ac9d72161b4/modules/rgbd/src/depth_cleaner.cpp#L240-L245

it does not delete an existing instance, memleak here

[andreaceruti]

@berak Thank you for your interest, I rushed a bit in writing the issue, I was reading the image depth with this flag cv2.IMREAD_ANYDEPTH, so it was reading a 16 bit single channel image. Also looking at image values with this command from imagemagick, I saw that there was a very slight difference between original image and cleaned image. The difference remains also If I run the methods 100 times on the same image. Thank you for the help, I will wait for any other experience with this method in this issue

[berak]

The difference remains also If I run the methods 100 times on the same image.

see Listing 1 in the paper for the idea behind it and indeed, the difference_threshold value is hardcoded to 10, that's why subsequent 99 more iterations wont improve it