Retreive depth from depthframe

[Zumbalamambo]

I'm using the following code to retrieve the depth point from the colour image pixel point (X,Y). I used this as a reference https://github.com/IntelRealSense/librealsense/issues/7754

public static Vector3 Map2DTo3D(this Intrinsics intrinsics, Vector2 pixel, float depth)
  {
     Vector3 point = new Vector3();
      float x = (pixel.x - intrinsics.ppx) / intrinsics.fx;
      float y = (pixel.y - intrinsics.ppy) / intrinsics.fy;
      if (intrinsics.model == Distortion.InverseBrownConrady)
      {
          float r2 = x * x + y * y;
          float f = 1 + intrinsics.coeffs[0] * r2 + intrinsics.coeffs[1] * r2 * r2 + intrinsics.coeffs[4] * r2 * r2 * r2;
          float ux = x * f + 2 * intrinsics.coeffs[2] * x * y + intrinsics.coeffs[3] * (r2 + 2 * x * x);
          float uy = y * f + 2 * intrinsics.coeffs[3] * x * y + intrinsics.coeffs[2] * (r2 + 2 * y * y);
          x = ux;
          y = uy;
      }
      point.x = depth * x;
      point.y = depth * y;
      point.z = depth;
      return point;
  }

I managed to retrieve the color intrinsic as follows,

colorCamIntrinsic = ActiveProfile.GetStream(Stream.Color).As<VideoStreamProfile>().GetIntrinsics();

Now what should be the depth value that needs to be passed to Map2DTo3D

[MartyG-RealSense]

Hi @Zumbalamambo In the issue that you linked to, one of the commenters at https://github.com/IntelRealSense/librealsense/issues/7754#issuecomment-1170684537 used the following definition for depth:

using (var depth = obj.Streams.FirstOrDefault(s => s.Stream == Stream.Depth && s.Format == Format.Z16).As<VideoStreamProfile>()) 

ResetMesh(depth.Width, depth.Height);

[Zumbalamambo]

@MartyG-RealSense May I know what is obj.Streams?

[Zumbalamambo]

@MartyG-RealSense i used the following code to retrieve the depth.

VideoStreamProfile depthVp = profile.Streams.FirstOrDefault(s => s.Stream == Stream.Depth && s.Format == Format.Z16) `` ?.As<VideoStreamProfile>()

But it is a VideoStreamProfile and not a float point value (depth). It's something else.

How do I retrieve the depth?

[MartyG-RealSense]

It looks as though you are using a script that originated from the Mapping 2D to 3D section of a C# / Unity guide at the link below (I have quoted this guide in cases on this forum in the past).

https://lightbuzz.com/intel-realsense-coordinate-mapping/

In the paragraph of that guide just before the script, they state that "the depth of the desired point can be found by calling the GetDistance() method of the DepthFrame class".

float depth = depthFrame.GetDistance(x, y);

[Zumbalamambo]

@MartyG-RealSense the retrieved XYZ data is totally wrong. Is there any example of click and know depth using Unity Realsense?

[Zumbalamambo]

Also, how do I flip the frame horizontally or vertically?

[MartyG-RealSense]

Unity RealSense example scenes render depth visually but do not have accompanying numeric depth values as an overlay. This is because the Unity wrapper converts the camera data into a Unity material so that it can be viewed on-screen as a graphical representation of the data.

Non-RealSense pointcloud generation tools in the Unity Asset Store are also typically visual-only with no display of the numeric values. For example:

https://assetstore.unity.com/packages/tools/utilities/point-cloud-viewer-and-tools-16019

The link below is a project that gives greater control over generating a pointcloud with RealSense data by using a Unity Visual Effect Graph, but again it does not return real-time numeric depth values.

https://github.com/keijiro/Rsvfx

---

In regard to flipping the orientation of the Unity RealSense material, I am not aware of a way to do so unfortunately. It is possible to perform a vertical flip of the orientation of the main camera image though by simply changing the Z rotation value of the main camera object from '0' to '180'.

If it were possible to flip the image instead of the camera then although the image would appear flipped, its calibration data and other parameters would not be flipped with it, as described by a RealSense team member at https://github.com/IntelRealSense/librealsense/issues/6023#issuecomment-597615585

[Zumbalamambo]

Feature Request:

An example to select a point in the Image and get the XYZ position via Unity.

[MartyG-RealSense]

I have forwarded your Unity example feature request to my Intel RealSense colleagues to seek their feedback. Thanks!

[MartyG-RealSense]

Hi @Zumbalamambo After discussion with my RealSense colleagues, an official feature request has been created for a Unity example program that performs the function of providing a depth value when the image is clicked on.

This case should be kept open whilst the feature request is active, but you do not need to take any further action. Thanks very much for the request!