nerfstudio-project
How to render data in different formats together?sunch as 3DGSresult with glb in nerfstudio-viser
Hello, I have added a flower in the 3DGS scene.flower is in glb format I think the data format of 3DGS is different from that of flowers, but they were successfully rendered together. Another example, I added an image to the 3DGS scene and successfully rendered them.
As far as I know, Viser is just a web page used to receive rendered images, and then Viser displays the results images on a web page. This may mean that the 3DGS result is rendered simultaneously with the flower, generating only one result image. I want to know the principle, can anyone help me? Thank you
Hello!
I think there's two possibilities here:
- If you're using Nerfstudio and the rendered Gaussians are sent using
set_background_image(), we have depth compositing support. There's an example here: https://github.com/nerfstudio-project/viser/blob/main/examples/17_background_composite.py - If you're rendering Gaussians by directly adding them via
add_gaussian_splats(), we should actually have correct alpha compositing because everything is placed into the same threejs scene.
Does that answer your question?
Hello!
I think there's two possibilities here:
- If you're using Nerfstudio and the rendered Gaussians are sent using
set_background_image(), we have depth compositing support. There's an example here: https://github.com/nerfstudio-project/viser/blob/main/examples/17_background_composite.py- If you're rendering Gaussians by directly adding them via
add_gaussian_splats(), we should actually have correct alpha compositing because everything is placed into the same threejs scene.Does that answer your question?
Thank you very much for your answer. I am based on the results of nerfstudio-splatfacto
And added a glb file (I am not familiar with the code, so I randomly added my code in the blank space of the code, luckily, it was successfully displayed)
For example, the PART2 in the picture should be a green branch, and now the code has successfully estimated the order of the front and back, implementing that the PART2 are behind the sculpture, the PART2 are not displayed, and the part1 and part3 are displayed normally.
I think the data format of 3DGS is different from that of flowers, but they were successfully rendered together. Meanwhile, the result graph successfully displays the occlusion sequence.
As far as I know, Viser is just a web page used to receive rendered images, and then Viser displays the results images on a web page. This may mean that the 3DGS result is rendered simultaneously with the flower, generating only one result image.
I want to know the principle.where is the code, can anyone help me?
Thank you
Hi @smart4654154, sorry for the late reply I've been on some deadlines!
Based on what you described this does look like just the depth compositing feature. When we send rendered images to Viser from nerfstudio we include a depth map:
https://github.com/nerfstudio-project/nerfstudio/blob/73fe54dda0b743616854fc839889d955522e0e68/nerfstudio/viewer/render_state_machine.py#L308-L313
The example I linked in the earlier comment is also still relevant. Here's also the shader that we use to write the depth map to the WebGL depth buffer:
https://github.com/nerfstudio-project/viser/blob/0cb94054b29028a012aeeba11b00a5477b09a2fa/src/viser/client/src/App.tsx#L576-L680
Hi @smart4654154, sorry for the late reply I've been on some deadlines!
Based on what you described this does look like just the depth compositing feature. When we send rendered images to Viser from nerfstudio we include a depth map:
https://github.com/nerfstudio-project/nerfstudio/blob/73fe54dda0b743616854fc839889d955522e0e68/nerfstudio/viewer/render_state_machine.py#L308-L313
The example I linked in the earlier comment is also still relevant. Here's also the shader that we use to write the depth map to the WebGL depth buffer:
viser/src/viser/client/src/App.tsx
Lines 576 to 680 in 0cb9405
/* Background image with support for depth compositing. */ function BackgroundImage() { // Create a fragment shader that composites depth using depth and rgb const vertShader = ` varying vec2 vUv;
void main() { vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0); }
.trim(); const fragShader =#includeprecision highp float; precision highp int; varying vec2 vUv; uniform sampler2D colorMap; uniform sampler2D depthMap; uniform float cameraNear; uniform float cameraFar; uniform bool enabled; uniform bool hasDepth;
float readDepth(sampler2D depthMap, vec2 coord) { vec4 rgbPacked = texture(depthMap, coord);
// For the k-th channel, coefficients are calculated as: 255 * 1e-5 * 2^(8 * k). // Note that: [0, 255] channels are scaled to [0, 1], and we multiply by 1e5 on the server side. float depth = rgbPacked.r * 0.00255 + rgbPacked.g * 0.6528 + rgbPacked.b * 167.1168; return depth;}
void main() { if (!enabled) { // discard the pixel if we're not enabled discard; } vec4 color = texture(colorMap, vUv); gl_FragColor = vec4(color.rgb, 1.0);
float bufDepth; if(hasDepth){ float depth = readDepth(depthMap, vUv); bufDepth = viewZToPerspectiveDepth(-depth, cameraNear, cameraFar); } else { // If no depth enabled, set depth to 1.0 (infinity) to treat it like a background image. bufDepth = 1.0; } gl_FragDepth = bufDepth;}`.trim(); // initialize the rgb texture with all white and depth at infinity const backgroundMaterial = new THREE.ShaderMaterial({ fragmentShader: fragShader, vertexShader: vertShader, uniforms: { enabled: { value: false }, depthMap: { value: null }, colorMap: { value: null }, cameraNear: { value: null }, cameraFar: { value: null }, hasDepth: { value: false }, }, }); const { backgroundMaterialRef } = React.useContext(ViewerContext)!; backgroundMaterialRef.current = backgroundMaterial; const backgroundMesh = React.useRef<THREE.Mesh>(null); useFrame(({ camera }) => { // Logic ahead relies on perspective camera assumption. if (!(camera instanceof THREE.PerspectiveCamera)) { console.error( "Camera is not a perspective camera, cannot render background image", ); return; }
// Update the position of the mesh based on the camera position. const lookdir = camera.getWorldDirection(new THREE.Vector3()); backgroundMesh.current!.position.set( camera.position.x, camera.position.y, camera.position.z, ); backgroundMesh.current!.position.addScaledVector(lookdir, 1.0); backgroundMesh.current!.quaternion.copy(camera.quaternion); // Resize the mesh based on focal length. const f = camera.getFocalLength(); backgroundMesh.current!.scale.set( camera.getFilmWidth() / f, camera.getFilmHeight() / f, 1.0, ); // Set near/far uniforms. backgroundMaterial.uniforms.cameraNear.value = camera.near; backgroundMaterial.uniforms.cameraFar.value = camera.far;});
return (
<planeGeometry attach="geometry" args={[1, 1]} /> ); }
Thank you very much. This is truly a genius idea, fantastic. https://github.com/nerfstudio-project/nerfstudio/blob/73fe54dda0b743616854fc839889d955522e0e68/nerfstudio/viewer/render_state_machine.py#L308-L313 the link is also the shader that we use to write the depth map to the WebGL depth buffer: The next step is in-depth testing To my knowledge, WebGL does not enable deep testing by default. As far as I know, to enable deep testing in WebGL, i need to call the gl.enable() method and pass gl.DEPTH_TEST as a parameter. I searched for gl.enable() in Viser, but couldn't find it. Could you share how you conduct in depth testing? Thank you