Returned depth images for certain scenes contain no data

[alt2014]

Following the unrealcv generate-images tutorial, I tried to get the depth images for the Infinity Blades: Grass Lands scene. However, the image I got back contained no data.

The depth image is supposed to be in the top righthand corner.

However, when I inspect the depth image inside Unreal, it is clear that there should be data here. The depth image is in the bottom left corner.

The tutorial example worked as expected for me, as did a few other scenes. I'm not sure what's different about this one.

[mikeroberts3000]

I am experiencing this problem also. The depth values returned for this scene are always 1.09. So I just get a 640x480 array of all 1.09 (or in other words, np.ones((480,640))*1.09). The tutorial examples and several other scenes work as expected. But the Infinity Blades: Grass Lands scene does not.

Is there some kind of screen space rendering effect I can turn off in the Unreal Editor to get working depth estimates for this scene? For my purposes, it doesn't matter if I make the scene look less realistic, because at this point I'm just trying to acquire depth. I tried going through the scene settings and disabling anything that seemed like it might turn on any kind of screen space rendering, but I still couldn't get depth working for this scene. That being said, I could easily be missing something obvious.

[qiuwch]

Hi, @alt2014 @mikeroberts3000

@alt2014 , In the BufferVisulization you saw, there are two types of depth, one is SceneDepth, another is SceneDepthWorldUnits. They are produced by post process material SceneDepth.uasset and SceneDepthWorldUnits.uasset respectively. You can use the UE4Editor to open these two files to see how the depth is computed. The visualization you saw is produced by SceneDepth.uasset.

I use the ·SceneDepthWorldUnits.uasset in UnrealCV, which depends on Camera Relative World Position information. I am still not sure why this info is not working for this specific map. I need to dig deeper to find out the reason.

I am traveling these days and will get back to you if I have more discovery.

[jason718]

@alt2014 @mikeroberts3000 @qiuwch Same problem here! But I could get the correct depth image in Editor. Then, I used 'vrun shot' to get the images. It seems like that the problem is in the 'vget ../depth', not in the depth model (SceneDepthWorldUnits.uasset).

[DonnEssime]

i have a similar issue, where vget /camera/0/depth would return completely different results from the ScenDepthWorldUnit buffer you get when dumping the buffer contents (with shot, after setting 'r.BufferVisualizationDumpFrames 1'). Some examples (from the PirateIsland marketplace assets, may exist with others as well)

The lit image: vget /camera/0/depth (note: similar view, but not quite): dumped SceneDepthWorldUnits:

note that in the unrealcv-captured depth, the box and stone are floating and the ground is completely flat. Also note the f'ed up plant depth. This is consistent, though, letting me create a point cloud out of the depth images:

NB: this is still the release unrealcv from september and UE 4.12, as the new one is currently compiling

[DonnEssime]

Update: this issue no longer occurs when using the current master branch against UE 4.14 (AND using the cnpy-interface rather than saving the image directly to disk, so i'm assuming that it's more a fact of the cnpy interface working differently)

[mikeroberts3000]

In case anyone is interested, I discovered that the SceneDepthWorldUnits material only renders sensible depth data when there is no screen-space post-processing happening. I was having the same issue as @alt2014 reported above, but I fixed it by disabling the PostProcessVolume object in the Infinity Blades: Grass Lands scene. After doing so, I was able to use UnrealCV to obtain meaningful depth data for this scene.

In particular, I selected the PostProcessVolume3 object in the World Outliner view of the Unreal Editor. Then I scrolled down to the Post Process Volume Settings in the Details pane for this object. Finally, I unchecked the "Enabled" checkbox. After making these changes, UnrealCV returned meaningful depth data.

Unfortunately, these changes also make the RGB images returned by UnrealCV noticeably darker than in the viewport of the Unreal Editor. For my use case, this is fine, because I can render RGB and depth images in two separate batches. A cleaner solution would probably be to programmatically disable all screen-space post-processing before attempting to render any depth images.

To debug this issue, I also found it useful to look at what the SceneDepth and SceneDepthWorldUnits materials do (File > Open Asset > View Options > Select "Show Engine Content" > Select SceneDepth or SceneDepthWorldUnits). Doing so demonstrates exactly how to interpret the values produced by the SceneDepthWorldUnits material, which is useful if you want to unproject them back into world coordinates.

[mikeroberts3000]

This issue appears to be present in the "4.22" branch built against Unreal 4.22.3.

Depth images do work correctly for Realistic Rendering, but they do not seem to work correctly for Infinity Blades: Grass Lands. My previously reported solution of disabling post-processing volumes doesn't seem to work any more. Remarkably, I also can't seem to get depth images working in a new project with no starter content (i.e., Epic Games Launcher > Launch > New Project > Select No Starter Content > Create Project).

[qiuwch]

It seems changing the GameMode to UnrealcvGameMode can solve it.

This is what I have.

[mikeroberts3000]

Hi @qiuwch, great to hear from you and to see all the fantastic progress on UnrealCV. I'm so glad I asked about this. Your fix worked for me. I also didn't need to manually disable the post-processing volumes in the scene like I did in 2017 😄

[qiuwch]

Great to see it works. This code is actively maintained, but the documentation is far behind, which I should spend more time on. We are actively using it in research projects of our team. If you saw more issues, please let me know and we might have a solution.