VLN-CE
VLN-CE copied to clipboard
jacobkrantz
questions about the VLNCE dataset
Hi, I feel quite confused about the dataset. In the vlnce gt dataset, there is locations and actions sequence, however when I use the simulator there is "vlnce-oricle action" in each step, and I found that this action sequence is different from the action sequecen in gt file。 Does someone know which one is the correct action sequence ?
Thank you for your question. Differences between the ground truth (GT) paths and paths generated from the VLNOracleActionSensor exist for two reasons (primarily the first):
- The GT path was constructed to travel within 0.5m of each position in the reference path (Matterport3D nav-graph nodes). The
VLNOracleActionSensorignores the reference path and provides the next action along the shortest path to the goal. This sensor was used in training as it enabled DAgger training and is a reasonable proxy for paths in the Room2Room (R2R) dataset; all R2R paths are shortest paths on the nav-graph. We have confirmed that GT paths are similar toVLNOracleActionSensorpaths with approximately a 0.96 SPL. - Both the GT dataset files and the
VLNOracleActionSensoruse the ShortestPathFollower from Habitat (the sensor uses it here). The GT path was constructed with a different version of Habitat's ShortestPathFollower than theVLNOracleActionSensorcurrently uses. If you want theVLNOracleActionSensorto use the same underlying oracle, one option is using Habitat-Lab v0.1.4 and a matching Habitat-Sim version. But that is a bandaid fix.
Thinking about reason 2) further, this discrepancy breaks training reproducibility of paper results since the paper used the v0.1.4 ShortestPathFollower. Also, the dataset path generation and pruning was performed using the old ShortestPathFollower. To fix this, we will bring back the old ShortestPathFollower in the VLN-CE code without downgrading the Habitat version.
Does someone know which one is the correct action sequence ?
The short answer is that the correct action sequence is the GT file.
ok. Thanks for answering. One more question, how do you make sure that the shortest path generated by the simulator will follow the language instruction ? It seems it is not necessary to follow the instruction, right ?
I also just found that there is "reference path", which is different from the location sequence in gt file and path generated by the sensor, in the dataset. I am wondering what is this "reference path“ and it is obtained ? Thanks in advanced.
Thank you for your question. Differences between the ground truth (GT) paths and paths generated from the
VLNOracleActionSensorexist for two reasons (primarily the first):
- The GT path was constructed to travel within 0.5m of each position in the reference path (Matterport3D nav-graph nodes). The
VLNOracleActionSensorignores the reference path and provides the next action along the shortest path to the goal. This sensor was used in training as it enabled DAgger training and is a reasonable proxy for paths in the Room2Room (R2R) dataset; all R2R paths are shortest paths on the nav-graph. We have confirmed that GT paths are similar toVLNOracleActionSensorpaths with approximately a 0.96 SPL.- Both the GT dataset files and the
VLNOracleActionSensoruse the ShortestPathFollower from Habitat (the sensor uses it here). The GT path was constructed with a different version of Habitat's ShortestPathFollower than theVLNOracleActionSensorcurrently uses. If you want theVLNOracleActionSensorto use the same underlying oracle, one option is using Habitat-Lab v0.1.4 and a matching Habitat-Sim version. But that is a bandaid fix.Thinking about reason 2) further, this discrepancy breaks training reproducibility of paper results since the paper used the v0.1.4 ShortestPathFollower. Also, the dataset path generation and pruning was performed using the old ShortestPathFollower. To fix this, we will bring back the old ShortestPathFollower in the VLN-CE code without downgrading the Habitat version.
Does someone know which one is the correct action sequence ?
The short answer is that the correct action sequence is the GT file.
ok. Thanks for answering. One more question, how do you make sure that the shortest path generated by the simulator will follow the language instruction ? It seems it is not necessary to follow the instruction, right ? I also just found that there is "reference path", which is different from the location sequence in gt file and path generated by the sensor, in the dataset. I am wondering what is this "reference path“ and it is obtained ? Thanks in advanced
The original VLN task is based on navigating from node to node on the Matterport3D graph. When we ported each trajectory to a continuous environment, we transferred not just the start and goal locations, but the intermediate node locations as well. The reference_path is this sequence of node locations from start to goal (including start and goal). The section "Transferring Nav-Graph Trajectories" in the paper has details of the process.
We ensure that the GT path matches reasonably with the language instruction by having the GT path travel within 0.5m of each position in the reference_path. The VLNOracleActionSensor does not guarantee alignment with the instruction -- there are occasional instances where the path generated by the VLNOracleActionSensor takes a different route, but this is empirically uncommon. Since all paths in the R2R dataset are shortest paths, this only happens when there is another approximate shortest path that happens to be shorter in continuous environments.
Many thanks for your answering. You mentioned that if I want to get the sensor that can generate same path as the path in gt file, I should use habitat-sim v0.1.4 and matching Habitat-sim. May I ask which version of Habitat-sim should I use, or , since you are also considering to bring the old follwer back, when will you get this done ?
The old follower is now back 🙂 (#8). If you want it to follow the reference path, you can do something like this: https://gist.github.com/jacobkrantz/9143f4744b9e1809bb1ef0522700932b
Hi, Have you cheked if the original follower can still genereate same action sequence ? I tried to use the origin follower but I still generate a difference action sequences.
Another question is that I used the action seqeunce in gt file to generate a location sequence, but then I found this location sequence is different from the location sequence in ft file.
@jacobkrantz , hI, i am sorry for bothering you agian. I found that I still get a differernt path when I used the old follower. Could you kindly check if the old follower could still generate the same path ?
Using Habitat v0.1.5 and ShortestPathFollowerCompat, I find that the GT action sequences match exactly for 95% of paths. The 5% of differences can be attributed to internal version differences in the simulator (same action results in slightly different location). If you want a perfect replication of the GT paths, you need to use the same versions of Habitat-Lab and Habitat-Sim used to generate them. I created a new branch that contains code compatible with these older versions as well as instructions for installation in the README: https://github.com/jacobkrantz/VLN-CE/tree/orig_habitat_versions. Using that Habitat version, I have confirmed for each episode that the generated action and location sequences match the GT paths. Where exact matching is not required, I recommend using the VLN-CE master branch.
Using Habitat v0.1.5 and
ShortestPathFollowerCompat, I find that the GT action sequences match exactly for 95% of paths. The 5% of differences can be attributed to internal version differences in the simulator (same action results in slightly different location). If you want a perfect replication of the GT paths, you need to use the same versions of Habitat-Lab and Habitat-Sim used to generate them. I created a new branch that contains code compatible with these older versions as well as instructions for installation in the README: https://github.com/jacobkrantz/VLN-CE/tree/orig_habitat_versions. Using that Habitat version, I have confirmed for each episode that the generated action and location sequences match the GT paths. Where exact matching is not required, I recommend using the VLN-CE master branch.
OK. I will try that. Many thanks for the explanation.