Question about the "SEMANTIC_SENSOR" and Camera Rotation

[Jthon-lab]

Habitat-Sim version

Habitat-Sim 0.2.2 Habitat-Lab 0.2.2

Docs and Tutorials

Did you read the docs? yes Did you check out the tutorials? yes

❓ Questions and Help

Question 1: How to get a first person view semantic images in Habitat-Lab or Habitat-Sim? I feed quite confused here because as I display the semantics results, it seems to be a global top-down view semantic images.

Question 2: In the ObjectNav task, the agent can execute the LookUp and LookRight actions, how to get the rotation matrix about the camera correctly? Here I try to use env._sim.get_agent_state().sensor_states['rgb'].position, env._sim.get_agent_state().sensor_states['rgb'].rotation to acquire them, but when I input the results to the function env._sim.get_observation_at(position,rotation), it gives a different images, especially when I execute LookUp or LookDown actions.

I paste the configuration files and the demo python codes, thanks for your reply!

ENVIRONMENT:
  MAX_EPISODE_STEPS: 500
SIMULATOR:
  TURN_ANGLE: 30
  TILT_ANGLE: 30
  ACTION_SPACE_CONFIG: "v1"
  AGENT_0:
    SENSORS: ['RGB_SENSOR', 'DEPTH_SENSOR', 'SEMANTIC_SENSOR']
    HEIGHT: 0.88
    RADIUS: 0.18
  HABITAT_SIM_V0:
    GPU_DEVICE_ID: 0
    ALLOW_SLIDING: False
  SEMANTIC_SENSOR:
    WIDTH: 256
    HEIGHT: 256
    HFOV: 79
    POSITION: [0, 0.88, 0]
  RGB_SENSOR:
    WIDTH: 256
    HEIGHT: 256
    HFOV: 79
    POSITION: [0, 0.88, 0]
  DEPTH_SENSOR:
    WIDTH: 256
    HEIGHT: 256
    HFOV: 79
    MIN_DEPTH: 0.5
    MAX_DEPTH: 5.0
    POSITION: [0, 0.88, 0]
TASK:
  TYPE: ObjectNav-v1
  POSSIBLE_ACTIONS: ["STOP", "MOVE_FORWARD", "TURN_LEFT", "TURN_RIGHT", "LOOK_UP", "LOOK_DOWN"]
  SENSORS: ['OBJECTGOAL_SENSOR', 'COMPASS_SENSOR', 'GPS_SENSOR']
  GOAL_SENSOR_UUID: objectgoal
  MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SUCCESS', 'SPL', 'SOFT_SPL']
  DISTANCE_TO_GOAL:
    DISTANCE_TO: VIEW_POINTS
  SUCCESS:
    SUCCESS_DISTANCE: 0.1

DATASET:
  TYPE: ObjectNav-v1
  SPLIT: val
  DATA_PATH: "/home/henry/Desktop/dataset/rl_envs/habitat-lab/data/datasets/objectnav/mp3d/v1/{split}/{split}.json.gz"
  SCENES_DIR: "/home/henry/Desktop/dataset/rl_envs/habitat-lab/data/scene_datasets/"

import habitat
import cv2
import os
import numpy as np
from habitat_sim.utils.common import d3_40_colors_rgb
os.environ["MAGNUM_LOG"] = "quiet"
os.environ["HABITAT_SIM_LOG"] = "quiet"
def make_semantic(semantic_obs):
    semantic_image = np.zeros((semantic_obs.shape[0],semantic_obs.shape[1],3),np.uint8)
    semantic_image = np.resize(d3_40_colors_rgb[semantic_obs.flatten()%40],semantic_image.shape)
    return semantic_image

config = habitat.get_config("./config/objectnav_mp3d.yaml")
env = habitat.Env(config=config)
observation = env.reset()
while not env.episode_over:
    camera_position = env._sim.get_agent_state().sensor_states['rgb'].position
    camera_rotation = env._sim.get_agent_state().sensor_states['rgb'].rotation
    while True:
        key = cv2.waitKey(1)
        if key == 97: #"a" = turn left
            action = 2 
            break
        elif key == 100: # "d" = turn right
            action = 3
            break
        elif key == 113:  # "q" = turn up
            action = 4
            break
        elif key == 101:  # "e" = turn down
            action = 5
            break
        elif key == 119:  # "w" = forward
            action = 1
            break
        cv2.imshow("rgb",observation['rgb'])
        cv2.imshow("camera_rgb",env._sim.get_observations_at(camera_position,camera_rotation)['rgb'])
        cv2.imshow("semantic",make_semantic(observation['semantic']))
    observation = env.step(action)

[aclegg3]

Looks like the semantic and RGB meshes may have different transforms. Did you use the most recent SceneDataset file as described in the dataset documentation?

[ghost]

Looks like the semantic and RGB meshes may have different transforms. Did you use the most recent SceneDataset file as described in the dataset documentation?

Thanks for your suggestion, I partially solved the Question 1, But I was wondering how to define the SceneDatasetConfig path in Habitat-Lab config file? I've tried setting DATASET.SCENE_DATASET but it doesn't work.

[aclegg3]

I've tried setting DATASET.SCENE_DATASET but it doesn't work.

It should be SIMULATOR.SCENE_DATASET because only the SIMULATOR sub-config group will be passed into the underlying habitat_simulator.py initialization routines. You can see all the available config options in habitat/config/default.py.