waymo-1137-not-wide not work

[chaoer]

Hi, Thanks for opensource your fancy work.

I tried the waymo-1137-not-wide.yaml for rendering efficiency, but result did not contain any object mentioned in the prompt. (even for the default prompt: "add a straight driving car in the scene" )

I notice that the rendering_mode is "render_hdr_shutter" in not-wide yaml. However, there is no mode named "render_hdr_shutter" in Exprunner file.

Is this the reason why no object appear in the final rendering result?

[yifanlu0227]

For some reason I'm just now getting around to replying to your issue, I'm very sorry.

You're right, while doing the code refraction we probably copied this outdated file, making render_hdr_shutter mode lost. You need to add these line to chatsim/background/mcnerf/src/ExpRunner.cpp and recompile McNeRF, then it should work

  ...
  else if (mode == "render_hdr_shutter"){
    RenderPath(true, true, false);
  }
  else {
    std::cout << "Unknown mode: " << mode << std::endl;
  }
  
}

[yifanlu0227]

We would fix this bug and make the repo accelerated and more user-friendly in the next several commits. Then you can fetch the latest code. I will reply in the issue again once they are done

[chaoer]

We would fix this bug and make the repo accelerated and more user-friendly in the next several commits. Then you can fetch the latest code. I will reply in the issue again once they are done

Thanks. Looking forward to your refactored code.

[chaoer]

BTW:

In current chatsim pipeline, we can obtain image in which a car occlude a tree if the car is before the tree. However, It seams that we can not get the tree occlude the car when the car is behind a tree.

Is there any possible that we render foreground object (e.g. car) with awareness of background depth (e.g. tree) in blender?

[yifanlu0227]

@chaoer Sorry for the late reply.

We leave the API for depth and occlusion detection (['agents']['foreground_rendering_agent']['estimate_depth/depth_est']), but we haven't found a robust detection estimation framework to calculate the background depth before. Previously we used LiDAR projected depth + SAM to determine the region-wise depth, but that's not a very robust solution.

We notice Metric3D makes things easier and we will try to incorporate them into our codebase.

[chaoer]

@yifanlu0227 Thanks for your continually refactor and improvement of this fancy work. I will close this issue, and looking forward to you coming improvements.