rgb after sim.step not changed

[yaofeng11]

Habitat-Sim version

v0.2.2 Sorry sir, I just use pygame.joystick to control robot. And my code is here, I wonder why I take per action, the state get new, this I just print it, but the observation just not change, I mean whether move_forward or turn_left/right, all the rgb is the same. do = False action_names = list(cfg.agents[sim_settings["default_agent"]].action_space.keys()) last_hat = (0, 0) last_orientation = 0 pygame.joystick.init() pygame.init()

map_frames = [] observations_list = [] while do==False: act = None # EVENT PROCESSING STEP for event in pygame.event.get(): # User did something if event.type == pygame.QUIT: # If user clicked close # Possible joystick actions: JOYAXISMOTION JOYBALLMOTION JOYBUTTONDOWN JOYBUTTONUP JOYHATMOTION if event.type == pygame.JOYBUTTONDOWN: print("Joystick button pressed.") if event.type == pygame.JOYBUTTONUP: print("Joystick button released.") if event.type == pygame.JOYHATMOTION: print(" Hat motion.")

joystick_count = pygame.joystick.get_count()
# For each joystick:
for i in range(joystick_count):
    joystick = pygame.joystick.Joystick(i)
    joystick.init()

hat = joystick.get_hat(0)
if hat[1] == 1 and last_hat[1] != 1:
    print("move_forward")
    act = action_names[0]
elif hat[0] == 1 and last_hat[0] != 1:
    print("turn_right")
    act = action_names[1]
elif hat[0] == -1 and last_hat[0] != -1:
    print("turn_left")
    act = action_names[2]
last_hat = hat
if act is not None:
    observations = sim.step(act)
    rgb = observations["color_sensor"]
    observations_list.append(sim.get_sensor_observations())
    print('display new frame')
    display_sample(rgb)
    new_state = agent.state
    agent_pos_new = new_state
    print(agent_pos_new)

I just want when I take per action, the camera would gain frames after each step. But I dont know why? Is there something wrong with my code?Thanks

❓ Questions and Help

[erikwijmans]

My guess is that the display_sample is where the bug is as there's no reason why the observation shouldn't be updating if the agent's position is.

[yaofeng11]

I just use the display_sample you give in the colab. def display_sample(rgb_obs): from habitat_sim.utils.common import d3_40_colors_rgb rgb_img = Image.fromarray(rgb_obs, mode="RGBA") arr = [rgb_img] titles = ["rgb"] plt.figure(figsize=(12, 8)) for i, data in enumerate(arr): ax = plt.subplot(1, 1, i + 1) ax.axis("off") ax.set_title(titles[i]) plt.imshow(data) plt.show()

[yaofeng11]

@erikwijmans Thanks for your quick reply. And I appreciate your help.Thanks. I use the hm3d val datasets. I mean if it possible that the problem is about scene test_scene = "/home/PycharmProjects/habitat/scene/scene_datasets/hm3d-val-habitat/00844-q5QZSEeHe5g/q5QZSEeHe5g.basis.glb"

Besides I just follow the random API you give in colab total_frames = 0 action_names = list(cfg.agents[sim_settings["default_agent"]].action_space.keys()) max_frames = 5 while total_frames < max_frames: action = random.choice(action_names) print("action", action) observations = sim.step(action) rgb = observations["color_sensor"] display_sample(rgb) total_frames += 1

That observations API may get the image after per action, and display_sample just define the way to show(use matplotlib)

[erikwijmans]

If the position is updating, the observation should update (there isn't anything scene specific that would change that).

Does the random API work as expected? It does for me.

My guess is that there is some weird interaction between habitat and pygame. I know there can be one since both habitat and pygame create OpenGL contexts (pygame can, not sure if what you are doing causes it to make one).

[yaofeng11]

@erikwijmans Thanks for your reply! I appreciate it. Yes, the random API just worked. I have already tested it. And besides I use another way to test the control.

count_step = 0
max_step = 100
action_names = list(cfg.agents[sim_settings["default_agent"]].action_space.keys())
max_frames = 10
map_frames = []
observations_list = []
while count_step < max_step:
    s = input('key = ')
    key = int(s)
    if key == 1:
        act = action_names[key-1]
        print("MoveAhead")
    elif key == 2:
        act = action_names[key-1]
        print("MoveLeft")
    elif key == 3:
        action_id = key - 1
        act = action_names[key-1]
        print("MoveRight")


    observations = sim.step(act)
    print(observations["color_sensor"])
    rgb = observations["color_sensor"]
    cv2.imshow("RGB", transform_rgb_bgr(observations["color_sensor"]))
    observations_list.append(transform_rgb_bgr(observations["color"]))
    display_sample(observations["color_sensor"])
    new_state = agent.state
    agent_pos_new = new_state
    print(agent_pos_new)
    count_step += 1

It works. May be your guess is right. But I don't know that I use pygame.joystick just for gain the input, for example, use xbox360 joystick to determine the input(like the code I show (observations = sim.step(act)) ) act. Is there any way to solve this [problem?] And by the way, if I use cv2.imshow("RGB", transform_rgb_bgr(observations["color_sensor"])), the cv2 window unable to load. Is it because the picture rendering speed is too slow？

[aclegg3]

Hey @yaofeng11, it's been awhile. Still having problems here? Otherwise I may close this issue.

[yaofeng11]

Hi @aclegg3 , I transfer the way of control, instead of joystick, for joystick, it still exists some problems related to the context it produced. Anyway thanks. And another question about multithreading. Is there any example for mulitiagent control? I mean when I use multithreading for multiagent control, it appears no context, I googled it and it says last sim is not cloesd. Is there any way to solve it?

[aclegg3]

Habitat-sim dies not currently support multiple instances in the same process. Instead of multi-threading you could try multi-process. This is how we currently do parallel rollout collection for RL in Habitat-lab.

We are planning to add some support for multi-agent (multiple agents coexisting in same environment) to Habitat-lab also. This is technically already supported in Habitat-sim as you can add multiple cameras or robots to the scene and control them simultaneously.

It sounds like we've resolved the original issues here or identified the limitations. If so, we can close this issue. If you have additional questions or bug reports, feel free to open specific issues for those.

[yaofeng11]

Thanks for your help @aclegg3