Incorrect Distance_to_goal metric in ObjectNav task

[jzhzhang]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: v0.2.1

Habitat-Sim: v0.2.1

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release versions of Habitat-Lab and Habitat-Sim? Your question may already be addressed in the latest versions. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Master branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-lab/ yes

Did you check out the tutorials? https://aihabitat.org/tutorial/2020/ yes

Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

I just build an ObjectGoal Env with Matterport3D semantic V0.1 and want to directly obtain some metrics from the environment to evaluate the performance of the agent. So I follow the instructions by setting the habitat config:

TASK: 
    TYPE: ObjectNav-v1 
    POSSIBLE_ACTIONS: ["STOP", "MOVE_FORWARD", "TURN_LEFT", "TURN_RIGHT", "LOOK_UP", "LOOK_DOWN"]
    SENSORS: ['GPS_SENSOR', 'COMPASS_SENSOR'] MEASUREMENTS: ['DISTANCE_TO_GOAL', 'SUCCESS', 'SPL'] 
    SUCCESS: 
        SUCCESS_DISTANCE: 0.2

Then I obtain the distance_to_goal, which I think is the geodesic distance to the nearest goal, by:

self._env.get_metrics()["distance_to_goal"]

However, the result is quite large while the agent is close to one of the target goals (And I believe the RGB image proves the distance to the goal should be small). It seems like it still points to a far-away goal.

metrics {'distance_to_goal': 17.821653366088867, 'success': 0.0, 'spl': 0.0, 'softspl': 0.27454757027106463}

BTW, the metric of starting point is:

metrics {'distance_to_goal': 26.245790481567383, 'success': 0.0, 'spl': 0.0, 'softspl': 0.0}

I do not know whether it is purposely designed as the distance to goal in ObjectNav. I hope my question is clear, and really appreciate any help.

[jzhzhang]

I further check the episode files in Matterport Semantic V0.1, like 4ok3usBNeis.json. And I find the goals in the episode is empty. Is there any connection to the above question?

"episode_id": "0", "scene_id": "hm3d/val/00877-4ok3usBNeis/4ok3usBNeis.basis.glb", "scene_dataset_config": "./data/scene_datasets/hm3d/hm3d_annotated_basis.scene_dataset_config.json", "additional_obj_config_paths": [], "start_position": [0.48757, -0.53553, 0.34791], "start_rotation": [0, 0.72799, 0, 0.68559], "info": {"geodesic_distance": 15.1179, "euclidean_distance": 9.79221, "closest_goal_object_id": 285}, "goals": [], "start_room": null, "shortest_paths": null, "object_category": "tv_monitor"

[erikwijmans]

Can you provided the episode ID and scene ID for an example with this issue?

The goals field being empty in the json is expected. That's gets populated on load using the goals_by_category field.

[jzhzhang]

The episodes_id is 39 and the scene_id is hm3d/val/00800-TEEsavR23oF/TEEsavR23oF.basis.glb The complete episode config is :

"episode_id": "39", "scene_id": "hm3d/val/00800-TEEsavR23oF/TEEsavR23oF.basis.glb", "scene_dataset_config": "./data/scene_datasets/hm3d/hm3d_annotated_basis.scene_dataset_config.json", "additional_obj_config_paths": [], "start_position": [1.95021, 0.16338, -5.66802], "start_rotation": [0, 0.98217, 0, 0.18802], "info": {"geodesic_distance": 26.24578, "euclidean_distance": 8.84655, "closest_goal_object_id": 51}, "goals": [], "start_room": null, "shortest_paths": null, "object_category": "toilet"}

[jzhzhang]

@erikwijmans Hi, I take a deep step into the code and data. Here is what i found:

The scene hm3d/val/00800-TEEsavR23oF/TEEsavR23oF.basis.glb only have one toilet target, and the weird distance_to_goal comes from the wrong current position or target position. Specifically, the starting position is [1.95021, 0.16338, -5.66802] and the target position is [-5.98071, 3.16338, -7.79924]. The euclidian distance is 8.8465557 which is almost the same as the 8.84655 in TEEsavR23oF.json. However, the starting position and target position do not share the same height.

[erikwijmans]

Looks like there are (at least) two toilets in the scene -- there's on upstairs also and that's the one with those target positions.

The one you show seems to be an annotation error or an issue with how we are coalescing labels (it's labeled as toilet seat).

We'll flag that.

[jzhzhang]

Thanks for your time. I will figure out another way to train my agent.

[jzhzhang]

@erikwijmans Hi, thanks for your kindy responce last time. I reopen this issue because I just found more weird results in the hm3d.

I try to evaluate my agent with hm3d validation set. Most of the time, the simulator gives me right evaluation, but there are many weird results.

distance_to_goal : 6.921640, softspl    : 0.000000, success    : 0.000000 

distance_to_goal : 2.276802, softspl    : 0.337738, success    : 0.000000

Really confused with the data, any help would be appreciated.