Support for fisheye image in Magnum

[matsuren]

🚀 Feature

Currently, only perspective images are available. Is it possible to add other camera models such as fisheye images or equirectangular images?

Motivation

Fisheye images and equirectangular images are also commonly used for robotics applications.

Additional context

From what I understood, it would be difficult to add fisheye or equirectangular models since the current rendering is based on rasterization. If the rendering is based on ray tracing, these camera models can be introduced as in here. (But, ray tracing is computationally expensive.) Please correct me if I am wrong.

Thank you,

[Skylion007]

Yeah doing fisheye properly does require ray tracing. However, there is a hack we can use to make pseudo-equirectangular images: @matsuren https://github.com/facebookresearch/habitat-lab/pull/478 demonstrates how to stitch an equi-rectangular image from 6 cubemaps (6 sensors in varying orientations).

This hack could run even faster if it were built into our a renderer via a shader so that it could take advantage of the built in cubemap shading hardware for GPUs instead of relying on CUDA compute cores.

[matsuren]

Thank you @Skylion007 ! I really appreciate it. I managed to produce equirectangular images by the following source code. I feel like there is an easy way to use obs_transformer, still this code get the job done.

import habitat
from habitat_baselines.common.baseline_registry import baseline_registry
from habitat_baselines.utils.common import batch_obs
import math
from copy import deepcopy
import matplotlib.pyplot as plt

config=habitat.get_config("configs/tasks/pointnav.yaml")
config.defrost()
CAMERA_NUM = 6
orient = [
    [0, math.pi, 0],  # Back
    [-math.pi / 2, 0, 0],  # Down
    [0, 0, 0],  # Front
    [0, math.pi / 2, 0],  # Right
    [0, 3 / 2 * math.pi, 0],  # Left
    [math.pi / 2, 0, 0],  # Up
]
sensor_uuids = []
# Setup six cameras, Back, Down, Front, Left, Right, Up.
if "RGB_SENSOR" in config.SIMULATOR.AGENT_0.SENSORS:
    config.SIMULATOR.RGB_SENSOR.ORIENTATION = orient[0]
    for camera_id in range(CAMERA_NUM):
        camera_template = f"RGB_{camera_id}"
        camera_config = deepcopy(config.SIMULATOR.RGB_SENSOR)
        camera_config.ORIENTATION = orient[camera_id]

        camera_config.UUID = camera_template.lower()
        sensor_uuids.append(camera_config.UUID)
        setattr(config.SIMULATOR, camera_template, camera_config)
        config.SIMULATOR.AGENT_0.SENSORS.append(camera_template)
config.freeze()

# Setup CubeMap2Equirec
obs_trans = baseline_registry.get_obs_transformer("CubeMap2Equirec")
cube2equirec = obs_trans(sensor_uuids, (256,512))

# Load embodied AI task (PointNav) and a pre-specified virtual robot
env = habitat.Env(
    config=config
)

observations = env.reset()

batch = batch_obs([observations])
trans_observations = cube2equirec(batch)
equirect = trans_observations['rgb_0'].numpy().squeeze()
plt.imshow(equirect)

So, if I want to simulate fisheye images, I need to implement something like Cube2Equirec for fisheye distortion model. Am I right?

FYI, I think I found a bug here. Don't you think it should be for camera_id in range(CAMERA_NUM):?

Thanks!

[matsuren]

For people who are interested in visualizing equirectangular images, check out this as well. you can move the position of the camera using "w", "a", and "d" keys.

[Skylion007]

FYI, I think I found a bug here. Don't you think it should be for camera_id in range(CAMERA_NUM):?

It's not a bug, it keeps the default RGB template to use its default values

[Skylion007]

For people who are interested in visualizing equirectangular images, check out this as well. you can move the position of the camera using "w", "a", and "d" keys.

This seems like it would be useful to abstract into a Habitat-Lab viewer class of some sort. If we interface in a generic way, it could even be used as a bridge to PyRobot.

[matsuren]

It's not a bug, it keeps the default RGB template to use its default values

I see. The default camera is facing backward.

[Skylion007]

#486 Mostly solves this problem, but we should look into actually incorporating this into the underlying Magnum library as a cubemap renderer so that it can take full advantage of the GPU and run even faster potentially.

Until then, we have pseudo-support for these types of sensors as ObservationTransforms in Habitat-Lab.

[matsuren]

Thanks! Check out #486 in habitat-lab for details if you're interested in generating fisheye images in Habitat-lab.