habitat-lab

habitat-lab copied to clipboard

Motivation and Context

This PR adds a set of simulator queries for geometric relationships between objects which may be leveraged by a high-level planner or to validate the results of environmental interactions. Also adds some debug line render utils to visualize the query results.

Current implementation is a work in progress prototype. Expect changing API surface, edge cases, and sub-optimal performance.

Currently supported queries:

• "above"/"below" - given an object, find all other objects which are below/above it via raycasting.
• "within" - given an object, find all other objects which contain it via raycasting.
• "ontop" - given a supporting object, find all other objects which are supported by it (i.e., they have active contact points with vertically oriented normals).
• "nearby" - in L2 and geodesic - get nearby objects
• "size_regularized_distance" - use object bounding boxes and relative displacement vector to get a heuristically regularized distance between objects.
  • Application: L2 distance between mixed large and small objects is not very accurate. For example, you can touch a sofa and have distance > 1 meter to it's "position". This function attempts to heuristically estimate distance using bounding boxes to trim the L2 distance for each object.

TODO:

• generalize "above" to other directional queries.
  • add relative directional queries (e.g. "right" or "left" of "objectB")
• integrate with clutter generator rejection sampling
• add more neighborhood queries such as "near"

Note: requires --bullet install.

How Has This Been Tested

TODO: add unit tests, documentation, etc...

For now, examples/viewer.py is added as an interactive sandbox/testbed and demo of the features. This viewer is a modifed version of the habitat-sim python viewer. Run it from habitat-lab root directory like:

python examples/viewer.py --dataset data/fpss/hssd-hab-articulated-uncluttered.scene_dataset_config.json --scene 102817140

or replace the dataset and scene with your own.

UI modifications (differences from default viewer):

• in "GRAB" mousemode (press 'm' to toggle):
  • SHIFT + click an object to set it to motion_type.DYNAMIC in order to move it around
  • click any object to select it, displaying the set of objects from the active query mode (text in the top-left corner shows this)
• 'k' to iterate through active query modes

Debug visualization color codes:

• yellow - the selected object
• green - ManagedRigidObjects which satisfy the query
• cyan - ArticulatedLinks which satisfy the query
• blue - ArticulatedObject parent bounding box if any link satisfies the query.

Example of "within" query: Example of "above" query:

Example of "ontop" query:

Example of size_regularized_distance:

• The length of the green line is the regularized distance. The yellow and blue lines show the trimmed distance from each object.

Example of "on_floor" query:

• The ring denotes heuristic vertical object size. Yellow is not on floor, green is on floor. The line shows displacement to navmesh from center. Green line segment is allowed distance for threshold and red is error.

https://github.com/facebookresearch/habitat-lab/assets/1445143/cfccdcd9-e863-4611-9907-08b8b68b982b

Example of "region" query:

• regions are drawn with heatmap based on containment percentage. Red region is primary.

https://github.com/facebookresearch/habitat-lab/assets/1445143/a2eb52ca-02f6-4b40-9747-d29d40e483bf

Types of changes

• [Development] A pull request that add new features to the habitat-lab task and environment codebase. Development Pull Requests must be small (less that 500 lines of code change), have unit testing, very extensive documentation and examples. These are typically new tasks, environments, sensors, etc... The review process for these Pull Request is longer because these changes will be maintained by our core team of developers, so make sure your changes are easy to understand!

Checklist

• [x] My code follows the code style of this project.
• [ ] I have updated the documentation if required.
• [x] I have read the CONTRIBUTING document.
• [x] I have completed my CLA (see CONTRIBUTING)
• [ ] I have added tests to cover my changes if required.

Heads up for those playing with this branch -- currently incompatible with hab sim nightly due to the magnum update. https://github.com/facebookresearch/habitat-lab/pull/1776

The relevant functionality from this prototype PR has been merged.