habitat-sim
habitat-sim copied to clipboard
facebookresearch
Assign mesh holes max depth for Depth Sensor
🚀 Feature
If there is a hole in the mesh during Depth rendering, assign max Depth for Depth Sensor output.
Motivation
Currently, mesh holes are assigned Depth equal to 0 for the Depth Sensor. On the pictures provided, there is a wall close to the camera. There are also the holes around cabinets that should have infinity depth, but their depth values are equal to 0 and are close to the wall in absolute value:
First, that makes a function of Depth to the distance from the camera nonmonotonic. That may affect models training or require post-processing on the Habitat Lab side. Second, when noise models have applied on top of the frame, it is impossible to distinguish holes from close objects.
Alternatives
Apply noise on Habitat Lab side with Observations Transform.
Additional context
That may affect trained baselines before, but that should be ok to do the change with version switch.
If there is a hole in the mesh during Depth rendering, assign max Depth for Depth Sensor output.
The decision to assign a hole to zero was/is intentional. That is what real depth sensors do for undefined values as it's impossible for things to actually be at a depth of zero.
I see, we going with emulating real Depth sensors here. @erikwijmans do you know if our current noise model have special handling of such undefined values? Thank you!
The depth noise model doesn't handle zero values differently than any other value, i.e. there is not a special case that guarantees that zero values in the ground truth depth are also zeros in the noisy depth. The code itself is fine with zeros however. Zeros that enter the part of the noise model that add noise come out as zeros, but there is a downsample and a pixel shuffle stage also, so that may lead to zeros becoming non-zero before noise is added.
Thank you for the clarification. Possibly, that can be a configurable feature of the depth sensor. Otherwise the same can be done with the observation transformation on Habitat Lab side with at batch scale.