Missing object meta after InitialRandomSpawn

[BoyuanChen]

Hi team,

I found that for certain environment and certain random seed, after InitialRandomSpawn action, there will be objects missing from the returned object metadata. One example to reproduce this issue is attached. I tested it on both 3.3.1 and 3.3.4.

After running the following script, the two print statements will print 38 and 37 respectively. The credit card object is missing.

Could you please help take a look at this? Thanks!!

import ai2thor.controller

scene = 'FloorPlan214_physics'
controller = ai2thor.controller.Controller(
    width=300,
    height=300,
    local_build=False,
    start_unity=True,
    scene=scene,
    port=8201,
    host='127.0.0.1',
    agentMode='default',
    )
event = controller.reset(scene)
object_meta = event.metadata['objects']
print(len(object_meta))


controller.reset(scene)
shuffled_event = controller.step(
    action='InitialRandomSpawn',
    randomSeed=17,
    forceVisible=False,
    numPlacementAttempts=5,
    placeStationary=True
    )

object_meta = shuffled_event.metadata['objects']
print(len(object_meta))

[mattdeitke]

Thanks @BoyuanChen!

I ran into this issue a long time ago with the rearrangement project, but was never able to reproduce it at the time. I can reproduce this, though!

Colab link: https://colab.research.google.com/drive/16QJjtq6p9Dje9_UGabLlLCiFhmQyG5-s?usp=sharing

[BoyuanChen]

Great! Thanks @mattdeitke !

Another side note that may be helpful is that I only encountered this when setting forceVisible=False. Out of 120 scenes, there were about 11 scenes that have this issue. However, I do not think this is scene-specific since changing random seed may or may not have this issue again.

[mattdeitke]

This is interesting to note on forceVisible.

Do you by chance have the 11 scenes? I can just run them forever with a bunch of random seeds in the background to just see if they're able to reproduce this issue.

I've previously just run InitialRandomSpawn 50 times, with different random seeds, in every scene to see if I was able to reproduce the issue, but that never resulted in finding any cases. Although, I might not have been using forceVisible=False.

Right now, we think the issue might have to do with an object having an incorrect bounding box annotation, rather than an issue with the InitialRandomSpawn action itself. So, it might be a bit of a manual updating process with a few outlying objects.

[Lucaweihs]

To be clear: is it that the object metadata is missing or that the object itself is missing (or not sure)?

[mattdeitke]

In the one example, a Plate in the scene breaks after calling InitialRandomSpawn (assumed by @winthos to be because of a bounding box issue). Then the credit card does not appear in the metadata, but it does appear in the scene. Having an object break and having an object missing from the metadata (that is not the broken object) is probably not a coincidence.

@winthos mentioned that it does appear in the metadata from the Unity side, but I don't see why there should be any disparities between Unity and Python here.

Still looking into it.

[winthos]

There is a chance that the Plate is not the only object type affected by this incorrect bounding box. It seems the CreditCard in 214 is also offset incorrectly. Because of this, I did notice in editor that the card would cause the sofa to slowly be pushed down into the floor. Is there any reason that an object out of scene bounds would not be included in metadata generation?

I've also made a branch with a fix for correctly zeroing out the ObjectBounds so that both the plate and the credit card, as well as other potentially affected objects, will now have the correct bounds: https://github.com/allenai/ai2thor/tree/ObjectBoundsFix

I'm waiting for that commit to build on that branch so I can check collab to see if fixing the bounding box issue also resolves the metadata. Mysteriously, I'm still not sure what is causing the metadata to skip over the credit card in the "broken" version currently on main.

[Lucaweihs]

@winthos I have code that will go through and fix all the bounding boxes in the prefabs, let me know if you'd like it (or it mostly does this, would need a small change to update the prefabs rather than just updating a single scene).

[BoyuanChen]

@mattdeitke Here are the 11 scenes.

'FloorPlan17_physics',
 'FloorPlan18_physics',
 'FloorPlan16_physics',
 'FloorPlan4_physics',
 'FloorPlan214_physics',
 'FloorPlan15_physics',
 'FloorPlan27_physics',
 'FloorPlan20_physics',
 'FloorPlan221_physics',
 'FloorPlan420_physics',
 'FloorPlan25_physics'

[winthos]

@winthos I have code that will go through and fix all the bounding boxes in the prefabs, let me know if you'd like it (or it mostly does this, would need a small change to update the prefabs rather than just updating a single scene).

We shouldn’t need this. Instead of updating prefab to prefab, we just account for potential offsets in the ObjectOrientedBoundingBox generation itself. It should have been something that was done from the beginning and this should work for all sim objects across the entire repo.

[winthos]

ok, i've confirmed all these issues were cascades off of each other. I've tried the same test collab matt threw together using the commit from my PR #821 and it correctly shows all 38 objects even after randomization, and the credit card is in scene.

https://colab.research.google.com/drive/1W9k6TPMOjD7k8vVdeXLhH3uB0u-Hu6Cy?usp=sharing

[BoyuanChen]

Thanks @winthos ! With your PR, I can also see the issue fixed on my side. If everything looks good to you all, I can go ahead and close the issue. Pls let me know. Thanks a lot for all of your helps!

[winthos]

@BoyuanChen Matt and I are doing some stress tests to see if there are other spawn related issues. I've got a few more corrections that I want to include with that PR, so that specific commit id might change in the next while here. I will update you again as we finalize the PR.

[mattdeitke]

FWIW, here are where the tests are running and failing still: https://colab.research.google.com/drive/1jysHfPA5NJdvjbTuXq1aWGK1fX4UNVeZ?usp=sharing