Merged procthor code

[jordis-ai2]

Merging procthor-rearrangement code into main.

[mariiak2021]

Hi @jordis-ai2, @Lucaweihs I'm looking forward to work with ai2thor-rearrangement challenge using PROCThor rooms dataset, can you please help me to understand how can I do so? Which branch can I use now? Can you please also share the example script to work with rearrangement one-phase or pwo-phase challenge using ProcTHOR dataset splits? Thank you a lot!

[jordis-ai2]

Hi @mariiak2021. It's currently not possible to use the ProcTHOR-related code base, since the underlying 10k houses dataset isn't yet publicly available. We're hoping to release everything soon, though, so you will be able to start training your own models in a very short time 👍

[mariiak2021]

Hi @jordis-ai2 thank you for the fast answer! Do you mean allenai/houses dataset? Is it based on procthor-10k dataset from prior package? If yes, can you please highlight the differences? It will help me a lot :) Can you also tell what is the expected date for allenai/houses to be released if it's known already? Thank you!

[jordis-ai2]

I think @mattdeitke or @Lucaweihs would have more accurate answers to these questions, but here's what I know: The currently missing dataset is composed of 10,000 ProcTHOR one and two-room houses, out of which we used 2,500 for our rearrangement experiments. This dataset is not yet available in our upcoming distribution package -as far as I know-. Once it is there and the ongoing PR is merged, our experiments should be repeatable. As mentioned above, this is planned for the next few weeks.

[mariiak2021]

Hi @jordis-ai2 , so sorry I've missed your reply! Thanx a lot for the update. :) Will wait the release, but in the meantime I played around with your code and created my own dataset from PROCThor-10K.

I wanted to use it within the rearrangement challenge, but got a problem with instance_detections2D dictionary. This returns for me now only high level objects without children. To be more precise I guess I can see those objects inside instance_detections2D, which were not not "objects" (f.ex. "wall" or "door"). Those objects, which are affected by those lines:

def fix_object_names(self, house):
        known_assets = defaultdict(int)
        to_traverse = house["objects"][:]
        while len(to_traverse):
            cur_obj = to_traverse.pop()
            cur_obj["id"] = f'{cur_obj["assetId"]}_{known_assets[cur_obj["assetId"]]}'
            known_assets[cur_obj["assetId"]] += 1
            if "children" in cur_obj:
                to_traverse.extend(cur_obj["children"][:])

        return house

don't appear at all in the instance_detections2D dictionary. Can you please suggest what can be the problem? Did I miss somewhere the necessary changes? Thank you!

[jordis-ai2]

Good job! In principle it should work just as well with those houses.

If I understand it correctly, the problem you describe is a mismatch between the objects available under metadata["objects"] (where, hopefully, the overridden IDs are actually shown) and metadata["instance_detections2D"].

I assume you initialized THOR with renderInstanceSegmentation=True (https://ai2thor.allenai.org/ithor/documentation/initialization/#initialization-renderinstancesegmentation).

I'm not sure which version of THOR you're using, but my guess is that instance segmentation might not be (fully) supported for ProcTHOR yet.

[mariiak2021]

Hi @jordis-ai2, To get instance_detections2D I'm using renderInstanceSegmentation=True and getting; controller.last_event.instance_detections2D as specified here https://ai2thor.allenai.org/ithor/documentation/environment-state/#event-instance_detections2d

In the metadata["objects"] the overridden IDs are shown: instead of 'id': 'Bed|2|0' we will see 'id': 'Bed_18_2_0' for example, which is the combination of 'assetId': 'Bed_18_2' + 0 as it's the only bed in the scene.

I have tried different versions of Ai2Thor with different errors:

1. Using THOR_COMMIT_ID = "90eac925dc750818890069e3131f899998dc58b4" and function procthor_reset with "CreateHouse" action, I'm getting such type of the error:

RuntimeError: NullReferenceException: Object reference not set to an instance of an object. trace:   at UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController+<>c__DisplayClass283_0.<CreateHouse>b__2 (System.String id) [0x00003] in <6afd8f78be764eeba7be30f178fa1cb8>:0
  at System.Linq.Enumerable+WhereEnumerableIterator`1[TSource].GetCount (System.Boolean onlyIfCheap) [0x0001c] in <351e49e2a5bf4fd6beabb458ce2255f3>:0
  at System.Linq.Enumerable.Count[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00029] in <351e49e2a5bf4fd6beabb458ce2255f3>:0
  at UnityStandardAssets.Characters.FirstPerson.BaseFPSAgentController.CreateHouse (Thor.Procedural.Data.ProceduralHouse house) [0x00088] in <6afd8f78be764eeba7be30f178fa1cb8>:0
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <695d1cc93cca45069c528c15c9fdd749>:0

2. Using THOR_COMMIT_ID = "391b3fae4d4cc026f1522e5acf60953560235971", "CreateHouse" is not available, so I'm suing simply self.controller.reset(scene=house) instead. In this case I'm able to get instance_detections2d dictionary, but only for general objects. F.ex. instead of getting the following keys:

window|2|1
Bed|2|0
wall|2|0.00|0.00|0.00|3.62
wall|2|0.00|0.00|7.23|0.00
room|2

I have only:

window|2|1
wall|2|0.00|0.00|0.00|3.62
wall|2|0.00|0.00|7.23|0.00
room|2

Here bed is missing, as it was inside "objects", which IDs were changed.

The funny thing is that if I don't use the fix_object_names function, instance_detections2D returns correct results. The code to reproduce work as expected (with commented out fix_object_names function) and wrong work:

!pip install --extra-index-url https://ai2thor-pypi.allenai.org ai2thor==0+391b3fae4d4cc026f1522e5acf60953560235971 &> /dev/null
!pip install --upgrade ai2thor-colab &> /dev/null
import ai2thor
import ai2thor_colab

from ai2thor.controller import Controller
from ai2thor_colab import (
    plot_frames,
    show_objects_table,
    side_by_side,
    overlay,
    show_video
)

ai2thor_colab.start_xserver()

"AI2-THOR Version: " + ai2thor.__version__
controller = Controller()
!pip install prior &> /dev/null
import prior
houses = prior.load_dataset("procthor-10k")
from collections import defaultdict
house = houses["train"][12]
def fix_object_names(house):
        known_assets = defaultdict(int)
        to_traverse = house["objects"][:]
        while len(to_traverse):
            cur_obj = to_traverse.pop()
            cur_obj["id"] = f'{cur_obj["assetId"]}_{known_assets[cur_obj["assetId"]]}'
            #cur_obj["name"] = f'{cur_obj["assetId"]}_{known_assets[cur_obj["assetId"]]}'
            known_assets[cur_obj["assetId"]] += 1
            if "children" in cur_obj:
                to_traverse.extend(cur_obj["children"][:])

        return house
house = fix_object_names(house)
controller.reset(scene=house, renderInstanceSegmentation=True)
event = controller.step(action="RotateLeft")
a = controller.last_event.instance_detections2D
for k in a.keys():
  print (k)
#print (controller.last_event.metadata["objects"])
from PIL import Image
Image.fromarray(controller.last_event.frame)

Do you have any ideas what can I do?

• F.ex is it possible to avoid IDs override? Can you explain why do we actually need this override? And is it must to override IDs specifically?
• Or instead can I change somehow the IDs of the objects passed to instance_detection2D frame? that they will be matched
• Or can I override IDs first when loading data from .pkl files to set up rearrangements, but then push IDs to their back condition to be able using instance_detection2D frames?

[jordis-ai2]

First of all, if the new IDs are used for objects but not for detections without any error message, it seems to be a THOR issue, which will probably disappear once ProcTHOR is officially released.

Please keep in mind you don't really need to access instance segmentation to solve the rearrangement task. Indeed, in the official challenge, the only available visual inputs are RGB and depth. I.e., for the intended scope of this PR, you can just leave the current ID override on (this is how our models were trained and evaluated).

It's possible that you don't need to override object IDs any longer. With the original procthor-10k and supporting THOR build, some object IDs appeared more than once in the same scene, leading to undesired behavior.

For the other questions, I'm afraid they're out of the scope of this PR.

I don't want to repeat myself, but, if you can wait for a few weeks, you will be able to use the official release, where things will be less likely to suddenly change. In any case, I think you're in the right direction 👍

[mariiak2021]

Hi @jordis-ai2 thank you for your answers! I really need instance_detection2D to get 2D bboxes for training my model. :) Anyway will wait then for the official release - thanx for the great job you are all doing!