Action to select metadata keys that are required

[Tushar-N]

A lot of metadata is sent over at each controller step, but not all this metadata is required.

1. Is it possible to set the scope of metadata returned by the server on the fly (as a controller action).
2. Will this make things faster because less data needs to be sent over, or is this not a speed bottleneck (compared to sending over frame/segmentation tensors etc)?

I could have sworn I saw something about this in an issue/pull request, but I can't find it now. Sorry if this is a duplicate.

---

For example, if I don't care about the agent, or scene bounds, or specific properties of objects.

controller.step(
  "UpdateRequiredMetadata",
  meta_keys={
    'agent': False, 
    'sceneBounds': False,
    'objects':{'axisAlignedBoundingBox': False, 'mass': False}
  }
)

or if I ONLY care about specific object types?

controller.step(
  "UpdateRequiredMetadata",
  obj_types={
    'Painting': False, 
    'DeskLamp': False,
  }
)

but later down the line, I change my mind and I do want info about one of them

controller.step(
  "UpdateRequiredMetadata",
  obj_types={
    'DeskLamp': True,
  }
)

[ekolve]

Yes, there is an action that you can run:

Pass a list of objectIds to the following action that you are interested in. This will greatly reduce the amount of metadata generated.

controller.step('SetObjectFilter', objectIds=[...])

If you want to remove the filter, call the following action.

controller.step('ResetObjectFilter')

[Tushar-N]

Perfect, looks like I missed this function. Thanks!

[Tushar-N]

Reopening this to say that it would be nice to have some extra functionality for this class of actions. For example:

• setting object filters based on objectType, rather than Id. Right now if I limit the scope to Knife and Lettuce, but then I slice the lettuce, I lose the ability to have the new sliced objects in the metadata.
• More fine-grained control over metadata keys (like I described in the original post)

[Tushar-N]

Also wanted to ask: If I limit objects in metadata, this also limits what objects I can interact with. For example this does not work, even though query.metadata["actionReturn"] contains the correct objectId.

# while standing right in front of a fridge
controller.step('SetObjectFilter', objectIds=['Lettuce|+00.22|+01.19|00.00'])
query = controller.step(action='GetObjectInFrame', x=0.5, y=0.5)
if query:
    controller.step(action='OpenObject', objectId=query.metadata["actionReturn"])

Is this intentional? My goal was to limit the volume of data exchanged by the server, not limit the agents interactions with these objects.

[mattdeitke]

SetObjectFilter should not affect any interactions -- it simply affects the metadata. So, I suspect that possibly the agent is blocking the fridge from opening fully, or something like that, which can be checked by printing the event summary:

# while standing right in front of a fridge
controller.step('SetObjectFilter', objectIds=['Lettuce|+00.22|+01.19|00.00'])
query = controller.step(action='GetObjectInFrame', x=0.5, y=0.5)
if query:
    event = controller.step(action='OpenObject', objectId=query.metadata["actionReturn"])
    print(event)

---

Here, try opening up to a small openness, as in openness=0.05, which is more likely to not be impacted by the agent:

# while standing right in front of a fridge
controller.step('SetObjectFilter', objectIds=['Lettuce|+00.22|+01.19|00.00'])
query = controller.step(action='GetObjectInFrame', x=0.5, y=0.5)
if query:
    event = controller.step(action='OpenObject', objectId=query.metadata["actionReturn"], openness=0.05)
    print(event)

[Tushar-N]

SetObjectFilter should not affect any interactions -- it simply affects the metadata

This doesn't seem to be the case in 2.7.4. I can run the same set of actions and successfully interact with objects by just commenting out the SetObjectFilter action. The problem is not restricted to just refrigerators.

[ekolve]

The ObjectFilter also limits what is considered during the visibility check, which is quite expensive. Visibility (aka interactability) is used to determine whether you should be able to interact with an object - in this case open it . Essentially, when you set the object filter you are indicating that you want to ignore all other objects. There are a few things you can do for your case. One is to call SetObjectFilter after GetObjectInFrame with a new list of objectIds (including the one you just received). The other would be to pass forceVisible=True to the OpenObject action.

As well, we plan on updating the behavior of the Filter to make it more flexible with respect to actions that receive objectIds that are potentially getting filtered out.

[Tushar-N]

The visibility check explains it. forceAction=True works for me, thanks. Calling SetObjectFilter after GetObjectInFrame doesn't work as well because I have to do this for every interaction, so it only helps for movement actions.

[ekolve]

The issue @Tushar-N identified where forceAction=True is required to perform actions such as OpenObject on objects not included in the filter has been resolved with PR #693.