Depth observation in viewer.py has overlapping with previous frames

[moyutianque]

Habitat-Sim version

v0.2.3

Habitat is under active development, and we advise users to restrict themselves to stable releases. Are you using the latest release version of Habitat-Sim? Your question may already be addressed in the latest version. We may also not be able to help with problems in earlier versions because they sometimes lack the more verbose logging needed for debugging.

Main branch contains 'bleeding edge' code and should be used at your own risk.

Docs and Tutorials

Did you read the docs? https://aihabitat.org/docs/habitat-sim/ yes Did you check out the tutorials? https://aihabitat.org/tutorial/2020/ yes Perhaps your question is answered there. If not, carry on!

❓ Questions and Help

I am playing with examples/viewer.py I added sim_settings['depth_sensor'] = True and use self.sim.get_sensor_observations() after each agent.act(x) to get depth observation and color observation. The color observation is correct without error, but it seems the buffer of depth observation is not cleared. There is a huge overlapping between the current frame and the previous frames. You can find the effect in the attached images.

[dhruvbatra]

That's odd. If you give us a minimal reproducible example script (with details of the scene), we can look at reproducing it.

[moyutianque]

Please find the testing data and script here just cd to the directory and

sh run_my_inter_replica.sh

Some of my previous outputs can be found in test_output

I run the script with Ubuntu 22.04 and purely conda installed version of habitat 0.2.3

This overlapping happens everytime I call self.sim.get_sensor_observations() line 467

[dhruvbatra]

Can we get a github repository where we can actually look at code?

[moyutianque]

sure I also uploaded it to https://github.com/moyutianque/habitat-viewer-test

[0mdc]

@moyutianque

This appears to be caused by the depth buffer not being cleared between draws.

Can you add this code immediately before self.sim.get_sensor_observations():

mn.gl.default_framebuffer.clear(mn.gl.FramebufferClear.COLOR | mn.gl.FramebufferClear.DEPTH)

[moyutianque]

@moyutianque

This appears to be caused by the depth buffer not being cleared between draws.

Can you add this code immediately before self.sim.get_sensor_observations():

mn.gl.default_framebuffer.clear(mn.gl.FramebufferClear.COLOR | mn.gl.FramebufferClear.DEPTH)

@0mdc Thanks for the suggestion. I tried to add this line in multiple places, but it seems the depth buffer is still not being cleared. It is strange that rgb and semantic observations do not have these overlapping effects. If possible, could you help to test the script on your machine? I am not sure whether it is problem outside this habitat-sim package.

[moyutianque]

Update, it seems without using from magnum.platform.glfw import Application as the interactive interface, the depth rendering does not have problems. I tried to keep settings the same but directly visualize observations by PIL.

[dhruvbatra]

CC: @mosra

[volcverse]

Hi. Have you guys solved this problem? Same problem with me and any response will be greatly appreciated.

[moyutianque]

Hi. Have you guys solved this problem? Same problem with me and any response will be greatly appreciated.

not solved. I give up using this interactive version

[volcverse]

I managed to "solve" this in a residual way.

if self.has_prev_depth:
         render_depth = obs['depth_sensor'] - self.prev_depth
else:
         self.has_prev_depth = True
         render_depth = obs['depth_sensor']
                
self.prev_depth = obs['depth_sensor']

And the render_depth is the depth value I get. Although this method doesnt tackle the root cause, it helps me get the depth observation.

[VladimirYugay]

@DummyNodeHead how did you run the viewer for the replica dataset? How to find the proper json file for that? For me it never works.

[volcverse]

@DummyNodeHead how did you run the viewer for the replica dataset? How to find the proper json file for that? For me it never works.

Hi. I met this problem before but I almost forget it now. Maybe you can show some details so I can help you. :)

Now this command works for me. ./build/viewer --dataset /PATH/TO/REPLICA/replica.scene_dataset_config.json -- frl_apartment_0

[mosra]

Sorry for being extremely late to reply here, the notifications fell through the cracks.

If the previous depth values are kept in there and clearing didn't help, it could be due to blending being accidentally kept enabled for depth unprojection, causing the values to be .. um, blended together.

I have no easy way to test myself but could you try doing this at the same location as the clear was suggested, i.e. before self.sim.get_sensor_observations()?

mn.gl.Renderer.disable(mn.gl.Renderer.Feature.BLENDING)

If this causes rendering artifacts for text / UI, you may want to mn.gl.Renderer.enable() it again after getting the observations.