I want to get depth image like Unreal sub window image

[4iue0]

My English Image is poor ,so it may be difficult to read.

I need to use accurate depth image to avoid collision. But I cannot get depth image like this. I usually get this.

I find I should change pixel_as_float = True and pfm file convert png file by using this python code. responses = client.simGetImages([airsim.ImageRequest("1", airsim.ImageType.DepthVis,True)]) response = responses[0] depth = np.array(response.image_data_float, dtype=np.float32) depth = depth.reshape(response.height, response.width) depth = np.array(depth * 255, dtype=np.uint8) cv2.imwrite('depth.png', depth)

This image is converted pfm file

I want to get depth image like Unreal sub window image. Please tell me accurate depth image.

[LSBOSS]

If you want to get accurate depth images to avoid collisions, I would not go with DepthVis but use DepthPerspective instead.

• DepthVis is clamping and colorizing all depth values between 0 and 100 m and assign them the gray values of 0 to 255 and it's mainly used for inspection by the human eye.

• DepthPerspective gives you the actual distance in meters when requested via float.

Example with Numpy and OpenCV:

import airsim
import numpy as np
import cv2

# Constants for visualization
MIN_DEPTH_METERS = 0
MAX_DEPTH_METERS = 100

# Connect to running UE4 instance
client = airsim.VehicleClient(timeout_value = 7200)
client.confirmConnection()

# Request DepthPerspective image as uncompressed float
response, = client.simGetImages(
    [
        airsim.ImageRequest("my_camera", airsim.ImageType.DepthPerspective, True, False),
    ]
)

# Reshape to a 2d array with correct width and height
depth_img_in_meters = airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height)
depth_img_in_meters = depth_img_in_meters.reshape(response.height, response.width, 1)

# Lerp 0..100m to 0..255 gray values
depth_8bit_lerped = np.interp(depth_img_in_meters, (MIN_DEPTH_METERS, MAX_DEPTH_METERS), (0, 255))
cv2.imwrite("depth_visualization.png", depth_8bit_lerped.astype('uint8'))

# Convert depth_img to millimeters to fill out 16bit unsigned int space (0..65535). Also clamp large values (e.g. SkyDome) to 65535
depth_img_in_millimeters = depth_img_in_meters * 1000
depth_16bit = np.clip(depth_img_in_millimeters, 0, 65535)
cv2.imwrite("depth_16bit.png", depth_16bit.astype('uint16'))

In the end, depth_visualization.png contains the visualization you asked for and depth_16bit.png is a more accurate depth map with values in millimetre range that you can use for your collision task.

I hope that helps you. Feel free to close that issue, if it resolves your question.

[rallen10]

I am facing a similar problem @LSBOSS thank you for the code, but it causes an error. at the line with list_to_2d_float_array, it errors with:

ValueError: cannot reshape array of size 1 into shape (144,256)

apparently response.image_data_float is a just a list [0.0] for some reason

[tallatmahmood98]

I am facing a similar problem @LSBOSS thank you for the code, but it causes an error. at the line with list_to_2d_float_array, it errors with:

ValueError: cannot reshape array of size 1 into shape (144,256)

apparently response.image_data_float is a just a list [0.0] for some reason

I am having the same issue. length of the image is 1, the height of the image 0, the width of image 0.

responses=client.simGetImages([airsim.ImageRequest("1", airsim.ImageType.DepthPerspective, True,False)]) 
img1d = np.array(responses[0].image_data_float, dtype=np.float)
img1d = 255/np.maximum(np.ones(img1d.size), img1d)
print('Size of img1d', len(img1d),img1d.size)
print('Height',responses[0].height,'width',responses[0].width)
img2d = np.reshape(img1d, (responses[0].height, responses[0].width))

this gives an image as expected but after some iterations, it gives an error that "cannot reshape array of size 1 to shape (0,0) "

[heleidsn]

Hi, @LSBOSS, thanks for your code. I find it's slow when I request DepthPerspective image using

airsim.ImageRequest("my_camera", airsim.ImageType.DepthPerspective, True, False)

It's about 5Hz for 400500 depth image. I even tried 10080 depth image, it's still 5Hz. Is this normal or do you have any idea to get DepthPerspective faster?

PS: I find if I set the pixels_as_float = False, it will be very fast (100Hz for 400*500 depth image), but I cannot recover the depth image from the binary string.

Many thanks.

Intel i7-8700 GTX1060 6G AirSim 1.3.1 UE 4.24.3

[jonyMarino]

I think #3018 would help

[ChangShenXu]

@LSBOSS Thank you very much for your codes. However, it doesn't seem to be able to acheive the effect as shown in "depth sub-window". I used to think the reason was the choice for the parameter MAX_DEPTH_METERS value. And I tried MAX_DEPTH_METERS = 51, 102, 153, 204, 255, the depth images for visualization were not improved too much. Sincerely hope for a better solution.

PS. Except for the path of saving the file, the following code is almost the same as that of LSBOSS.

import airsim
import numpy as np
import cv2
import tempfile
import os

# Constants for visualization
MIN_DEPTH_METERS = 0
MAX_DEPTH_METERS = 100

# Connect to running UE4 instance
#client = airsim.VehicleClient(timeout_value = 7200)
#client.confirmConnection()
client = airsim.MultirotorClient()
client.confirmConnection()
client.enableApiControl(True)
client.armDisarm(True)

# Request DepthPerspective image as uncompressed float
response, = client.simGetImages(
    [
        airsim.ImageRequest("0", airsim.ImageType.DepthPerspective, True, False),
    ]
)

# Reshape to a 2d array with correct width and height
depth_img_in_meters = airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height)
depth_img_in_meters = depth_img_in_meters.reshape(response.height, response.width, 1)

tmp_dir = os.path.join(tempfile.gettempdir(), "airsim_drone")
# Lerp 0..100m to 0..255 gray values
depth_8bit_lerped = np.interp(depth_img_in_meters, (MIN_DEPTH_METERS, MAX_DEPTH_METERS), (0, 255))
filename = os.path.join(tmp_dir, 'depth_visualization')
print("png filename is %s.png" % filename)
cv2.imwrite(os.path.normpath(filename + '.png'), depth_8bit_lerped.astype('uint8'))
print(depth_8bit_lerped)

# Convert depth_img to millimeters to fill out 16bit unsigned int space (0..65535). Also clamp large values (e.g. SkyDome) to 65535
depth_img_in_millimeters = depth_img_in_meters * 1000
depth_16bit = np.clip(depth_img_in_millimeters, 0, 65535)
filename = os.path.join(tmp_dir, 'depth_16bit')
filename = os.path.normpath(filename + '.png')
cv2.imwrite(filename, depth_16bit.astype('uint16'))

client.armDisarm(False)
client.reset()

# that's enough fun for now. let's quit cleanly
client.enableApiControl(False)

[h-gokul]

I too have the same issue, Is there a solution for this ?

[godhj93]

If you want to get accurate depth images to avoid collisions, I would not go with DepthVis but use DepthPerspective instead.

• DepthVis is clamping and colorizing all depth values between 0 and 100 m and assign them the gray values of 0 to 255 and it's mainly used for inspection by the human eye.
• DepthPerspective gives you the actual distance in meters when requested via float.

Example with Numpy and OpenCV:

import airsim
import numpy as np
import cv2

# Constants for visualization
MIN_DEPTH_METERS = 0
MAX_DEPTH_METERS = 100

# Connect to running UE4 instance
client = airsim.VehicleClient(timeout_value = 7200)
client.confirmConnection()

# Request DepthPerspective image as uncompressed float
response, = client.simGetImages(
    [
        airsim.ImageRequest("my_camera", airsim.ImageType.DepthPerspective, True, False),
    ]
)

# Reshape to a 2d array with correct width and height
depth_img_in_meters = airsim.list_to_2d_float_array(response.image_data_float, response.width, response.height)
depth_img_in_meters = depth_img_in_meters.reshape(response.height, response.width, 1)

# Lerp 0..100m to 0..255 gray values
depth_8bit_lerped = np.interp(depth_img_in_meters, (MIN_DEPTH_METERS, MAX_DEPTH_METERS), (0, 255))
cv2.imwrite("depth_visualization.png", depth_8bit_lerped.astype('uint8'))

# Convert depth_img to millimeters to fill out 16bit unsigned int space (0..65535). Also clamp large values (e.g. SkyDome) to 65535
depth_img_in_millimeters = depth_img_in_meters * 1000
depth_16bit = np.clip(depth_img_in_millimeters, 0, 65535)
cv2.imwrite("depth_16bit.png", depth_16bit.astype('uint16'))

In the end, depth_visualization.png contains the visualization you asked for and depth_16bit.png is a more accurate depth map with values in millimetre range that you can use for your collision task.

I hope that helps you. Feel free to close that issue, if it resolves your question.

It works well, but querying float data is very slow... (0.4sec/image) Is there any solution with low latency?