how to add the depth map from colmap to instant-ngp ?

[heyikou]

Is it correct to modify transform.json in the following way to add depth information? Why is the model worse when I raise depth_supervision_lambda in the GUI? By the way, I have multiple camera internal parameters to run colmap2nerf.py

{ "aabb_scale": 32, "enable_depth_loading": true, "integer_depth_scale": 1, "frames": [ { "camera_angle_x": 1.0400947633055835, "camera_angle_y": 0.7325250916894537, "fl_x": 5239.033097711955, "fl_y": 5214.176783015801, "k1": -0.15389681096543817, "k2": 0.13989445429774383, "p1": 0.0013421141111874388, "p2": -0.0006926718621671958, "cx": 3008.207391286385, "cy": 2040.88752627426, "w": 6000.0, "h": 4000.0, "aabb_scale": 16, "file_path": "./0503_20/5/SHEN0504S00122.JPG", "depth_path": "./depth_maps_png/5/SHEN0504S00122.png", "sharpness": 392.2510455343065, "transform_matrix": [ [ -0.0006291177692614196, -0.4863317119983009, 0.8737740383198666, 2.5107925386425918 ], [ 0.9999910204040466, -0.003967824929768948, -0.0014884477150855326, 0.35970091926335074 ], [ 0.00419086173772796, 0.8737652557731416, 0.4863298411651963, 2.0499815033639552 ], [ 0.0, 0.0, 0.0, 1.0 ] ] },

And the depth map looks like (between 0~255):

[meriemjabri]

did you find an answear ? I would also like to know how to add a depth map into instant-ngp

[jc211]

Are you sure your integer_depth_scale is correct? This is the number that is multiplied by your depth map to transform it into world units (the units your camera poses were taken in). If this number is very wrong, then the depth supervision would be making your reconstruction worse not better.

[ddkats]

Is it correct to modify transform.json in the following way to add depth information?

Why is the model worse when I raise depth_supervision_lambda in the GUI? By the way, I have multiple camera internal parameters to run colmap2nerf.py { "aabb_scale": 32, "enable_depth_loading": true, "integer_depth_scale": 1, "frames": [ { "camera_angle_x": 1.0400947633055835, "camera_angle_y": 0.7325250916894537, "fl_x": 5239.033097711955, "fl_y": 5214.176783015801, "k1": -0.15389681096543817, "k2": 0.13989445429774383, "p1": 0.0013421141111874388, "p2": -0.0006926718621671958, "cx": 3008.207391286385, "cy": 2040.88752627426, "w": 6000.0, "h": 4000.0, "aabb_scale": 16, "file_path": "./0503_20/5/SHEN0504S00122.JPG", "depth_path": "./depth_maps_png/5/SHEN0504S00122.png", "sharpness": 392.2510455343065, "transform_matrix": [ [ -0.0006291177692614196, -0.4863317119983009, 0.8737740383198666, 2.5107925386425918 ], [ 0.9999910204040466, -0.003967824929768948, -0.0014884477150855326, 0.35970091926335074 ], [ 0.00419086173772796, 0.8737652557731416, 0.4863298411651963, 2.0499815033639552 ], [ 0.0, 0.0, 0.0, 1.0 ] ] },

And the depth map looks like (between 0~255):

Did you find an answer to use the depth maps?

[poyodiaz]

@heyikou hello i am trying to solve this too ...did you find the way to do it? i tried the same and my project doesnt load at all, thanks in advance

[jexiaong]

I would like to know too if anyone has any updates ^.^

[poyodiaz]

@jexiaong i did this https://github.com/NVlabs/instant-ngp/discussions/647 initially my files didnt load then I didnt used the "integer depth scale" line in the transforms .json and the file loaded but i am not pretty sure if instant NGP is using the info for NErf construction,(seems like it is loading the path to the depth maps) i had to switch to another things and havent tried to much since then...

[ddkats]

@jexiaong i did this #647 initially my files didnt load then I didnt used the "integer depth scale" line in the transforms .json and the file loaded but i am not pretty sure if instant NGP is using the info for NErf construction,(seems like it is loading the path to the depth maps) i had to switch to another things and havent tried to much since then...

Did you enable depth supervision value in the GUI while training? My thought is that depth maps serve as a ground truth for minimizing the error of final nerf model, but im not sure. When I enable depth supervision, the result seems to be worse (enters black spots)... Another issue to discuss is the format of the depth images, I mean as far as I can understand ingp expects a float 16bit format in pixels, normalized at [0,1]. Is that correct? @heyikou any suggestions?

[jexiaong]

@jexiaong i did this #647 initially my files didnt load then I didnt used the "integer depth scale" line in the transforms .json and the file loaded but i am not pretty sure if instant NGP is using the info for NErf construction,(seems like it is loading the path to the depth maps) i had to switch to another things and havent tried to much since then...

Did you enable depth supervision value in the GUI while training? My thought is that depth maps serve as a ground truth for minimizing the error of final nerf model, but im not sure. When I enable depth supervision, the result seems to be worse (enters black spots)... Another issue to discuss is the format of the depth images, I mean as far as I can understand ingp expects a float 16bit format in pixels, normalized at [0,1]. Is that correct? @heyikou any suggestions?

By enable depth supervision, do you mean the "depth supervision strength" slider? Or is there another option in the GUI or code that I have to change? So far the only changes I've made was to the transform.json, and when I move the "depth supervision strength" slider my image becomes completely illegible

[poyodiaz]

@jexiaong i did this #647 initially my files didnt load then I didnt used the "integer depth scale" line in the transforms .json and the file loaded but i am not pretty sure if instant NGP is using the info for NErf construction,(seems like it is loading the path to the depth maps) i had to switch to another things and havent tried to much since then...

Did you enable depth supervision value in the GUI while training? My thought is that depth maps serve as a ground truth for minimizing the error of final nerf model, but im not sure. When I enable depth supervision, the result seems to be worse (enters black spots)... Another issue to discuss is the format of the depth images, I mean as far as I can understand ingp expects a float 16bit format in pixels, normalized at [0,1]. Is that correct? @heyikou any suggestions?

By enable depth supervision, do you mean the "depth supervision strength" slider? Or is there another option in the GUI or code that I have to change? So far the only changes I've made was to the transform.json, and when I move the "depth supervision strength" slider my image becomes completely illegible

@jexiaong i did move the depth supervision slider and did not see any change in the reconstruction , while you say that you changed your json what is exactly what you changed? could you please share ? I added the depth map for each frame and added the line "enable_depth_loading": true, but if i add "integer_depth_scale": 1,it doesnt loads at all and it crashes the app.

maybe my depth images are not generated correctly?

[jexiaong]

This is what my image looks like without depth And when I move the depth slider to 1 it becomes completely illegible (this is the same scene and viewing direction) and this is what my transforms.json looks like with a depth path for each image I was wondering if it was because my depth images are png? I saw in this discussion https://github.com/NVlabs/instant-ngp/discussions/647 someone was using a .npy instead so should I be converting it? Or to some other format?

[ddkats]

@jexiaong > >

This is what my image looks like without depth And when I move the depth slider to 1 it becomes completely illegible (this is the same scene and viewing direction) and this is what my transforms.json looks like with a depth path for each image I was wondering if it was because my depth images are png? I saw in this discussion #647 someone was using a .npy instead so should I be converting it? Or to some other format?

I have the same problem when I load the depth files. Could you please share an example of your depth PNG? I don't think it's necessary to use .npy. I believe the issue may be related to the format of the depth PNG files (normalized to [0-1] or [0-255]?, pixels or meters?, float 16bit or 8bit?).

Regarding your earlier question about the depth supervision slider GUI, yes, that's what I meant.

[jexiaong]

@DimitrisKatsatos This is the depth image for the current project I'm working on, i ran it a few times on diff file formats ^this jpeg is 8bit (i think) and it didn't work ^this is the png of the same depth data 16bit (i think) and it also doesn't work ^this is an rgb png and it also doesn't work

[poyodiaz]

@jexiaong i did this #647 initially my files didnt load then I didnt used the "integer depth scale" line in the transforms .json and the file loaded but i am not pretty sure if instant NGP is using the info for NErf construction,(seems like it is loading the path to the depth maps) i had to switch to another things and havent tried to much since then...

Did you enable depth supervision value in the GUI while training? My thought is that depth maps serve as a ground truth for minimizing the error of final nerf model, but im not sure. When I enable depth supervision, the result seems to be worse (enters black spots)... Another issue to discuss is the format of the depth images, I mean as far as I can understand ingp expects a float 16bit format in pixels, normalized at [0,1]. Is that correct? @heyikou any suggestions?

@DimitrisKatsatos I have an issue that i think you guys dont , if i use the line : "integer depth scale : 1," on my json transforms file my app crashes and the project doesnt loads at all , does something comes to your mind about what could be causing this behaviour, my png files seems to be 16 bit normalized to 0,1

[poyodiaz]

one more question, when you guys load your files does the cmd line window of instatn NGP shows the same number of images loaded if you add the depth path or shall it be d showing the number of images multiplied by 2 (number of imges + number of depth images)?

[poyodiaz]

Ifound out that if i set : "enable_depth_loading": true, "integer_depth_scale": 1,

my files doest loads because of this error

[jexiaong]

Ifound out that if i set : "enable_depth_loading": true, "integer_depth_scale": 1,

my files doest loads because of this error

I had this error for one of my datasets too and I found out that if I delete a few images from training set then it will run. Make sure to limit the number of images ur training on to around 150 or less and that your images aren't too similar in angle (ensure that there's a noticeable change in angle between two consecutive pictures)

Also regarding your last question about the number of images, could you indicate which line exactly you're talking about? I didn't look too closely at the code, I'm not seeing where exactly in the cmd line is indicating the number of pictures?

[poyodiaz]

Ifound out that if i set : "enable_depth_loading": true, "integer_depth_scale": 1, my files doest loads because of this error

I had this error for one of my datasets too and I found out that if I delete a few images from training set then it will run. Make sure to limit the number of images ur training on to around 150 or less and that your images aren't too similar in angle (ensure that there's a noticeable change in angle between two consecutive pictures)

Also regarding your last question about the number of images, could you indicate which line exactly you're talking about? I didn't look too closely at the code, I'm not seeing where exactly in the cmd line is indicating the number of pictures?

@jexiaong i just found out it only happens if i reload a file with the app running , but if i close Instant NGP an re open it each time I load a new project i dont have the error and my projects loads ok ,after this , now i have the same behaviour you have when i turn on the slider of depth supervision on GUI (bad reconstruction) , .I suspect it could be the depth maps are inverted (white vs black values) if i set the slider to negative 1the issue disapears , ill keep trying and let you know

[meriemjabri]

max_depth= np.max(image_generated) uint16_array = (image_generated / max_depth * 65535).astype(np.uint16) quantized_img = Image.fromarray(uint16_array) quantization_scale = max_depth / 65535 #this should be integer_depth_scale value

this was my approach to generate the depth map. Still didn't get any improvement. How did you generate the depth map ? Btw you also need to add this lines in run.py testbed.render_mode = ngp.RenderMode.Depth testbed.nerf.training.depth_supervision_lambda = 1.0

[ddkats]

testbed.render_mode = ngp.RenderMode.Depth testbed.nerf.training.depth_supervision_lambda = 1.0

I use something like this for the depth maps:

cvimg = cv_bridge.imgmsg_to_cv2(msg) cvimg = cv2.normalize(cvimg, None, 0, 1, cv2.NORM_MINMAX, dtype=cv2.CV_32F) cvimg = (cvimg * 65535).astype(np.uint16)

I also tried your snippet (via PIL), but it outputs the same result. The fact is, the quality of the reconstruction is bad when depth maps are included. I don't know what seems to be the problem. In the transform.json you used integer_depth_scale=$value where value=max_depth / 65535?

[jexiaong]

this was my approach to generate the depth map. Still didn't get any improvement. How did you generate the depth map ?

Are you trying to implement depth supervision or generate a depth map? If you're trying to generate a depth map in the GUI then change the render mode to depth and set the exposure to the minimum. If you're talking about supervision then ignore me

[ddkats]

Are you trying to implement depth supervision or generate a depth map? If you're trying to generate a depth map in the GUI then change the render mode to depth and set the exposure to the minimum. If you're talking about supervision then ignore me

Thanks, but I 'm talking about depth supervision

[jexiaong]

Sorry I meant @meriemjabri

In the transform.json you used integer_depth_scale=$value where value=max_depth / 65535?

Also I used value=max_depth / 65535 as well but it didn't work. I'll try editing the run.py file though, I did not touch that previously

[poyodiaz]

In the transform.json you used integer_depth_scale=$value where value=max_depth / 65535?

@DimitrisKatsatos how should i implement this in the transforms.jason? i would like to try

[meriemjabri]

@poyodiaz if you have points cloud value you can project them into 2D image to get the depth map @DimitrisKatsatos yes integer_depth_scale=$value where value=max_depth / 65535 but it may be wrong since the value will be normalized and muliplying it by 65535 will cause overflow. Check this https://github.com/NVlabs/instant-ngp/issues/939 specially the last comment.

[jexiaong]

I don't know why this worked and it's not exactly precise but I can finally see my bear's 3D model! I initially had integer_depth_scale = max_depth / 65535 = 2299 / 65535 = 0.035080491340505077 (2299 is in mm) which wasnt working with depth supervision but then I arbitrarily multiplied by 0.1 such that integer_depth_scale = 0.0035080491340505077 and for some reason this worked. I don't exactly know what the math behind this is or whether it works for all datasets or just this one, but if anyone has any ideas please let me know

[ddkats]

I don't know why this worked and it's not exactly precise but I can finally see my bear's 3D model! I initially had integer_depth_scale = max_depth / 65535 = 2299 / 65535 = 0.035080491340505077 (2299 is in mm) which wasnt working with depth supervision but then I arbitrarily multiplied by 0.1 such that integer_depth_scale = 0.0035080491340505077 and for some reason this worked. I don't exactly know what the math behind this is or whether it works for all datasets or just this one, but if anyone has any ideas please let me know

Hi @jexiaong,

What do you mean this worked? Are you able to enable depth maps and improve the error with depth supervision? Or do you just mean that you manage to load the depth maps? However, I also tried what you said about integer_depth_scale = max_depth / 65535 = and multiply by 0.1 but it didn't work, the result is obviously worse than without using depth.

Regarding the current setup after enabling depth supervision, can you see black spots in the output? I'm not sure if there's a problem with the depth maps themselves (maybe they're too noisy) or their format.

[jexiaong]

@DimitrisKatsatos I'm unsure about how to check what the error is with each render could you guide me on how to do that? But yes I was able to load the depth maps and the results are comparable to the no-depth version visually Also this integer_depth_scale = max_depth / 65535 = and multiply by 0.1 has worked for me on two sets of data already (where the depth maps are in mm, I don't know how it would work with other depth img formats) however the *0.1 is by no means an accurate number cuz I still have no idea where it came from ^.^

Also what do you mean by black spots? Can you make out your figure with the addition of black spots or can you not see your figure at all?

[kafei123456]

@DimitrisKatsatos I'm unsure about how to check what the error is with each render could you guide me on how to do that? But yes I was able to load the depth maps and the results are comparable to the no-depth version visually Also this integer_depth_scale = max_depth / 65535 = and multiply by 0.1 has worked for me on two sets of data already (where the depth maps are in mm, I don't know how it would work with other depth img formats) however the *0.1 is by no means an accurate number cuz I still have no idea where it came from ^.^

Also what do you mean by black spots? Can you make out your figure with the addition of black spots or can you not see your figure at all?

Hello ，I'm confusing that the depth map must be complete (no any loss depth Or error depth points)?

[kafei123456]

one more question, when you guys load your files does the cmd line window of instatn NGP shows the same number of images loaded if you add the depth path or shall it be d showing the number of images multiplied by 2 (number of imges + number of depth images)? Hi guy, this is my loading process, I not sure whether Depht map be loaded successfully.