Depth image normalization seems to loose details

[cansik]

I am currently trying to capture both, the color image and the depth image. This is working, but the depth image looks really bad. The nearer areas are all blown out, sometimes the whole image is white.

The depth stream instead looks like how the death image should look like. I use the same method to convert the CVPixelBuffer into a grayscale image (the one you are using in the example).

I am missing something there, or why are these depth images not normalised the same way? I also tried to capture a depth image with the Halide App and they seem much better in depth resolution then the one I can capture / stream with lumina. Are there other ways to capture depth images or normalize them?

Here is the code I use to normalize the depth images:

func captured(stillImage: UIImage, livePhotoAt: URL?, depthData: Any?, from controller: LuminaViewController) {
	// save color image
	CustomPhotoAlbum.shared.save(image: stillImage)

	// save depth image if possible
	print("trying to save depth image")
	if #available(iOS 11.0, *) {
	    if var data = depthData as? AVDepthData {
	        
	        // be sure its DisparityFloat32
	        if data.depthDataType != kCVPixelFormatType_DisparityFloat32 {
	            data = data.converting(toDepthDataType: kCVPixelFormatType_DisparityFloat32)
	        }

	        guard let depthImage = data.depthDataMap.normalizedImage(with: controller.position) else {
	            print("could not convert depth data")
	            return
	        }

	        ...

extension CVPixelBuffer {
    func normalizedImage(with position: CameraPosition) -> UIImage? {
        let ciImage = CIImage(cvPixelBuffer: self)
        let context = CIContext(options: nil)
        if let cgImage = context.createCGImage(ciImage, from: CGRect(x: 0, y: 0, width: CVPixelBufferGetWidth(self), height: CVPixelBufferGetHeight(self))) {
            return UIImage(cgImage: cgImage , scale: 1.0, orientation: getImageOrientation(with: position))
        } else {
            return nil
        }
    }
    
    private func getImageOrientation(with position: CameraPosition) -> UIImageOrientation {
        switch UIApplication.shared.statusBarOrientation {
        case .landscapeLeft:
            return position == .back ? .down : .upMirrored
        case .landscapeRight:
            return position == .back ? .up : .downMirrored
        case .portraitUpsideDown:
            return position == .back ? .left : .rightMirrored
        case .portrait:
            return position == .back ? .right : .leftMirrored
        case .unknown:
            return position == .back ? .right : .leftMirrored
        }
    }
}

Here are the three images:

Color Image

Depth Image from the captured method

Depth image from the streamed method

[cansik]

Ok, it seems that there is a difference between relative and absolute depth buffers. The relative one is only for foreground / background separation, while absolute is to measure real distances. Is it possible to tell lumina, which one I would like to get?

[dokun1]

Hey @cansik - thanks for your work here. This is illuminating and I think this will have to be fixed.

I think this is something that I'll have to build in. Sounds like getting a still photo defaults to relative and streaming defaults to absolute, so we'll have to make a set of options agnostic of whether the data is being streamed or captured still.

I have quite a busy few days ahead of me, but can you find and show me where these settings are made? I think this would be a good feature to add to v1 of this.

[dokun1]

Hey @cansik just letting you know that I finally have some time to take a look at this - I'll be in Italy all week this week on vacation but I'll see what I can come up with on my flights!

[diospyros]

I think you just need to normalize the buffer. Add this to CVPixelBuffer:

extension CVPixelBuffer { func normalize() {

  let width = CVPixelBufferGetWidth(self)
  let height = CVPixelBufferGetHeight(self)

  CVPixelBufferLockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))
  let floatBuffer = unsafeBitCast(CVPixelBufferGetBaseAddress(self), to: UnsafeMutablePointer<Float>.self)

  var minPixel: Float = Float.greatestFiniteMagnitude
  var maxPixel: Float = -minPixel

  for y in 0 ..< height {
     for x in 0 ..< width {
    let pixel = floatBuffer[y * width + x]
 	    minPixel = min(pixel, minPixel)
  	    maxPixel = max(pixel, maxPixel)
     }
  }
  let range = maxPixel - minPixel

   for y in 0 ..< height {
 	    for x in 0 ..< width {
        let pixel = floatBuffer[y * width + x]
	    	floatBuffer[y * width + x] = (pixel - minPixel) / range
   }
   }
  CVPixelBufferUnlockBaseAddress(self, CVPixelBufferLockFlags(rawValue: 0))

} }

And in your function normalizedImage(), add this:

self.normalize()

right before this:

let ciImage = CIImage(cvPixelBuffer: self)