Problems in displaying the RGB observation

[jsdd25]

❓ Questions and Help

Hello. I found the <Habitat-sim Basics for Navigation tutorial> is the one I should follow.

Then I create a tiny version of the tutorial by copying some parts of the tutorial on my desktop as follows:

test_scene = "/home/robot/code/habitat/gibson_habitat/gibson/Pablo.glb"
rgb_sensor = True  # @param {type:"boolean"}
depth_sensor = True  # @param {type:"boolean"}
semantic_sensor = True  # @param {type:"boolean"}

sim_settings = {
    "width": 256,  # Spatial resolution of the observations
    "height": 256,
    "scene": test_scene,  # Scene path
    "default_agent": 0,
    "sensor_height": 1.5,  # Height of sensors in meters
    "color_sensor": rgb_sensor,  # RGB sensor
    "depth_sensor": depth_sensor,  # Depth sensor
    "semantic_sensor": semantic_sensor,  # Semantic sensor
    "seed": 1,  # used in the random navigation
    "enable_physics": False,  # kinematics only
}


def make_cfg(settings):
     sim_cfg = habitat_sim.SimulatorConfiguration()
     sim_cfg.gpu_device_id = 0
     sim_cfg.scene_id = settings["scene"]
     sim_cfg.enable_physics = settings["enable_physics"]

     sensor_specs = []
     color_sensor_spec = habitat_sim.CameraSensorSpec()
     color_sensor_spec.uuid = "color_sensor"
     color_sensor_spec.sensor_type = habitat_sim.SensorType.COLOR
     color_sensor_spec.resolution = [settings["height"], settings["width"]]
     color_sensor_spec.position = [0.0, settings["sensor_height"], 0.0]
     color_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
     sensor_specs.append(color_sensor_spec)

     depth_sensor_spec = habitat_sim.CameraSensorSpec()
     depth_sensor_spec.uuid = "depth_sensor"
     depth_sensor_spec.sensor_type = habitat_sim.SensorType.DEPTH
     depth_sensor_spec.resolution = [settings["height"], settings["width"]]
     depth_sensor_spec.position = [0.0, settings["sensor_height"], 0.0]
     depth_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
     sensor_specs.append(depth_sensor_spec)

     semantic_sensor_spec = habitat_sim.CameraSensorSpec()
     semantic_sensor_spec.uuid = "semantic_sensor"
     semantic_sensor_spec.sensor_type = habitat_sim.SensorType.SEMANTIC
     semantic_sensor_spec.resolution = [settings["height"], settings["width"]]
     semantic_sensor_spec.position = [0.0, settings["sensor_height"], 0.0]
     semantic_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE
     sensor_specs.append(semantic_sensor_spec)

     agent_cfg = habitat_sim.agent.AgentConfiguration()
     agent_cfg.sensor_specifications = sensor_specs
     agent_cfg.action_space = {
          "move_forward": habitat_sim.agent.ActionSpec(
               "move_forward", habitat_sim.agent.ActuationSpec(amount=0.25)
          ),
          "turn_left": habitat_sim.agent.ActionSpec(
               "turn_left", habitat_sim.agent.ActuationSpec(amount=30.0)
          ),
          "turn_right": habitat_sim.agent.ActionSpec(
               "turn_right", habitat_sim.agent.ActuationSpec(amount=30.0)
          ),
     }

     return habitat_sim.Configuration(sim_cfg, [agent_cfg])


cfg = make_cfg(sim_settings)

sim = habitat_sim.Simulator(cfg)

random.seed(sim_settings["seed"])
sim.seed(sim_settings["seed"])

agent = sim.initialize_agent(sim_settings["default_agent"])
agent_state = habitat_sim.AgentState()
agent_state.position = np.array([-0.6, 0.0, 0.0])  # world space
agent.set_state(agent_state)

agent_state = agent.get_state()
print("agent_state: position", agent_state.position, "rotation", agent_state.rotation)

total_frames = 0
action_names = list(cfg.agents[sim_settings["default_agent"]].action_space.keys())

max_frames = 5

while total_frames < max_frames:
     action = random.choice(action_names)
     print("action", action)
     observations = sim.step(action)
     rgb = observations["color_sensor"]
     semantic = observations["semantic_sensor"]
     depth = observations["depth_sensor"]
     rgb_img = Image.fromarray(rgb.astype(np.uint8), mode="RGBA")
     rgb_img = rgb_img.convert('RGB')
     rgb_img.save("test.jpeg")
     total_frames += 1

The RGB observation cannot be displayed and is completely black! Then, I tried most scenes in gibson, I found that RGB observations cannot be displayed normally in most scenes and only can be displayed normally in a few scenes.

Also, when I replace the test_scene by the example scene "17DRP5sb8fy.glb", the RGB observation can be displayed normally.

Could anyone give any hint on this problem? Any suggestion would be highly appreciated.

[erikwijmans]

agent_state.position = np.array([-0.6, 0.0, 0.0])

This demo is placing the agent in a fixed location. This location won't be appropriate for all scenes. You can use agent_state.position = sim.pathfinder.get_random_navigable_point() to set it to a reasonable location for that scene.

[jsdd25]

Hi @erikwijmans ,

Thanks so much for your help. I changed the localization using agent_state.position = sim.pathfinder.get_random_navigable_point(). However, I found that the RGB observation still cannot be displayed and is completely black in some Gibson dataset, such as "Scioto.glb" and "Colebrook.glb". The RGB observation can be displayed normally in "Pablo.glb" of Gibson dataset and the example scene "17DRP5sb8fy.glb"

Would you mind giving me any hint? Thanks.

[erikwijmans]

I cannot reproduce this. Here's a position that works in Scioto for me: [-9.643689 3.125416 1.8010855]

[HomeworldL]

I meet the same problem when using Gibson datasets. Most RGB images in Gibson datasets are black. Only a few datasets can be used, such as Eudora.glb. I test it in Ubuntu 20 04, python 3. 7 and the latest habitat-sim. Have you solved this problem?

[erikwijmans]

As far as we know this isn't a reproducible issue, so no we haven't fixed it. What type of GPU are you running habitat-sim with? Maybe there's a specific edge case in OpenGL with lower VRAM GPUs that is causing this.

[HomeworldL]

Thanks for your reply. I use a desktop with dual graphics cards:

• NVIDIA Corporation tu106 [geforce RTX 2060 Rev.A]
• Mesa Intel UHD Graphics 630(CML GT2)

The habitat-sim runs in the conda environment with RTX 2060. Maybe it cause by the version of OpenGL?

[erikwijmans]

Maybe. A 2060 should have enough vram to load a Gibson scene without needing a stream textures, but it's suspicious that it's only some scenes that have this issue.

[HomeworldL]

This problem might happen when install habitat on machines with multiple GPUs. It can be solved by using the installation option given in README.md. conda install habitat-sim withbullet headless -c conda-forge -c aihabitat

[swami23]

I am having same issues with RTX4090

[Yingxiu-Chang]

I got same issues. Does anyone solve it?

[vcozzolino]

I also have the same problem (blank RGB) with an RTX4080. Does it matter if I'm running in headless mode (I suppose not)?