rgb image quality in gibson scene data

[Roberyan]

trafficstars

Habitat-Sim version

v0.2.1

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release version of Habitat-Sim? Your question may already be addressed in the latest version. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Main branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-sim/

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/

Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

I want to extract the surrounding images from the scene in gibson data, but I find the quality of image is really poor, either can not tell what the objects really are or even with large black area. I wonder why this happens, is this normal or glb data related issue? I wonder could it be my code is doing the sampling by randomly select and set position and angles, therefore the simulation hasn't completed the rendering?

Below is my core code after simulation, setting of my camera is image_width=1024, image_height=1024, camera_height=1.5, camera_hfov=79:

# Load Habitat config
        cfg = get_habitat_config_V2(scene_glb_path)
        sim = habitat_sim.Simulator(cfg)
        if os.path.exists(scene_navmesh_path):
            sim.pathfinder.load_nav_mesh(scene_navmesh_path)
            print(f"Using NavMesh: {scene_navmesh_path}")

        # Initialize the agent
        agent = sim.get_agent(0)

        with h5py.File(scene_sem_map_path, 'r') as fp:
            floor_ids = sorted([key for key in fp.keys() if is_int(key)])
            for floor_id in floor_ids:
                name = f'{scene_name}_{floor_id}'
                map_world_shift = maps_info[scene_name]['map_world_shift']
                map_y = maps_info[scene_name][floor_id]['y_min']
                resolution = maps_info[scene_name]['resolution']
                map_semantic = np.array(fp[floor_id]['map_semantic'])

                tmp_scene_save_dir = os.path.join(TMP_SAVE_DIR, name)
                os.makedirs(tmp_scene_save_dir, exist_ok=True)

                # Convert to one-hot if needed (here we use the original sem_map)
                # semmap_oh = convert_maps_to_oh(map_semantic, dset=dset)

                # --- Compute available (navigable) area ---
                nav_coords_px = get_navigable_area_coordinates(map_semantic, floor_label=FLOOR_ID)
                if not nav_coords_px:
                    print(f"No navigable pixels found for {name}")
                    continue
                
                print(f"{len(nav_coords_px)} navigable positions")

                # Get boundaries (in world coordinates) of the navigable area:
                boundaries = get_navigable_area_boundaries(map_semantic, resolution, np.array(map_world_shift), map_y, floor_label=FLOOR_ID)
                if boundaries is not None:
                    world_min, world_max = boundaries
                    print(f"{name} navigable area boundaries (world coords):")
                    print(f"   Min: {world_min}")
                    print(f"   Max: {world_max}")
                else:
                    print(f"Could not determine boundaries for {name}")

                 # Optionally, convert a sampled pixel to world coordinate:
                
                for _ in range(5): # get 100 points
                    sampled_pixel = random.choice(nav_coords_px)
                    
                    # Sample a heading from multiples of 30 degrees.
                    sampled_angle = random.choice(range(0, 360, 30))
                    print(f"{name} sampled navigable pixel {sampled_pixel} with direction {sampled_angle}°")
                    
                    sample_save_dir = os.path.join(tmp_scene_save_dir, f"location_{sampled_pixel}_direction_{sampled_angle}")
                    os.makedirs(sample_save_dir, exist_ok=True)

                    map_img_with_sampled_point = visualize_sem_map(map_semantic, selected_point=sampled_pixel, selected_angle=sampled_angle)
                    cv2.imwrite(f"{sample_save_dir}/topdown_map.png", map_img_with_sampled_point)
                    
                    # --- try get the corresponding position in habitat sim and extract image
                    sampled_world_coord = pixel_to_world(sampled_pixel, resolution, np.array(map_world_shift), map_y)
                    print(f"{name} sampled navigable pixel {sampled_pixel} -> world position {sampled_world_coord}")


                    agent_state = agent.get_state()

                    habitat_angle = -(sampled_angle+90)%360

                    sampled_quat = common_utils.quat_from_angle_axis(
                        np.deg2rad(habitat_angle), 
                        np.array([0, 1, 0])
                    )
                    agent_state.position = np.array(sampled_world_coord)
                    agent_state.rotation = sampled_quat
                    agent.set_state(agent_state)
                    print("Agent placed at:", agent.get_state().position)
                    print("Agent rotation (quaternion):", agent.get_state().rotation)

                    # --- Capture surrounding images from that position ---
                    rgb_images = []
                    for rel_angle in [0, 90, 180, 270]:
                        # Compute the absolute angle (map & simulation) for this view.
                        abs_angle = (sampled_angle + rel_angle) % 360
                        habitat_angle = -(abs_angle+90)%360
                        agent_state = agent.get_state()

                        rotation = common_utils.quat_from_angle_axis(np.deg2rad(habitat_angle), np.array([0, 1, 0]))
                        agent_state.rotation = rotation
                        agent.set_state(agent_state)
                        
                        obs = sim.get_sensor_observations()
                        rgb_img = obs["color_sensor"]
                        rgb_bgr = cv2.cvtColor(rgb_img, cv2.COLOR_RGB2BGR)
                        rgb_images.append(rgb_bgr)
                        
                        cv2.imwrite(f"{sample_save_dir}/View_{abs_angle:.0f}.png", rgb_bgr)
                        print(f"Captured view at absolute angle: {abs_angle:.0f}°")
                    
                    panorama = np.hstack(rgb_images)
                    cv2.imwrite(f"{sample_save_dir}/Panorama.png", panorama)

        sim.close()