Problem with the real world data

[paulpacaud]

Thank you for your work, and sharing the repo.

i am reaching out for clarification: The real world dataset seems inconsistent for some tasks. Let's look at the task heatPotato2 in the metadata stages, it is written there are 17 stages but looking at your json file, there are actually 19 subactions. Moreover, it is mentioned that the failure reason is "The robot failed because it attempted to open the microwave door while it was still toggled on", but the failure arises at 01:48, which is the time when the robot tries to toggle on the microwave, not to open it, which is inconsistent with the failure reason. Another mysterious thing is the presence of the actions "Skip" and "Ignore" which are explained nowhere in the codebase, but still correspond to step frames of the video.

"Task 30": {
    "task_idx": 4,
    "name": "serve a bowl of potato on table that was heated using a microwave",
    "general_folder_name": "heatPotato2",
    "gt_failure_reason": "The robot failed because it attempted to open the microwave door while it was still toggled on.",
    "gt_failure_step": "01:48",
    "object_list": ["microwave with a white casing", "dark blue bowl", "orange table", "potato"],
    "actions": [
        "Pick up potato",
        "Put potato in bowl",
        "Skip",
        "Ignore",
        "Toggle on microwave",
        "Skip",
        "Ignore",
        "Open microwave",
        "Pick up bowl",
        "Put bowl in microwave",
        "Close microwave",
        "Skip",
        "Ignore",
        "Open microwave",
        "Pick up bowl",
        "Put bowl on table",
        "Terminate",
        "Close microwave",
        "Terminate"
    ],
    "success_condition": "a bowl of a heated potato is on the table."
}

1. I am kindly asking the authors if they could clarify what are the Skip, Ignore actions
2. how to deal with the discrepancy of action count in the json and stages count in the data ?

Thanks !