Pending after computing MSE and PSNR

[hzhshok]

Hello, Recently i made myself test data to practically check what resolution and parameters is suitable for my desktop, i made the resolution of 1264x1264, and last it pended even for 24 hours, i checked it consumed the approximate 7408g GPU, so, is my data too rough(MSE - 0.2591xx, last img_loss=1.109134) or the other reason?

So, someone give me a share/suggestion about how to do to my high resolution please? e.g. the relationship, 11g GPU: max resolution and scale.

In addition: actually my images is very high resolution(3000x4000), but to my GPU(3080 11g), i scale that to 1264x1264 after some tries.

Hardware: DESKTOP RTX 3080(11g), cuda 11.3. Software: ubuntu20.04 Running GPU cost during pending: always fixed cost, approximate 7.xg Running output and strace log: see the following.

The console output: python3 train.py --config ./configs/nerf_handong.json Config / Flags:

config ./configs/nerf_handong.json iter 2000 batch 1 spp 1 layers 4 train_res [1264, 1264] display_res [1264, 1264] texture_res [2048, 2048] display_interval 0 save_interval 100 learning_rate [0.03, 0.01] min_roughness 0.08 custom_mip False random_textures True background white loss logl1 out_dir out/nerf_handong ref_mesh data/nerf_synthetic/handong base_mesh None validate True mtl_override None dmtet_grid 128 mesh_scale 2.75 env_scale 1.0 envmap None display [{'latlong': True}, {'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}] camera_space_light False lock_light False lock_pos False sdf_regularizer 0.2 laplace relative laplace_scale 3000 pre_load True kd_min [0.0, 0.0, 0.0, 0.0] kd_max [1.0, 1.0, 1.0, 1.0] ks_min [0, 0.1, 0.0] ks_max [1.0, 1.0, 1.0] nrm_min [-1.0, -1.0, 0.0] nrm_max [1.0, 1.0, 1.0] cam_near_far [0.1, 1000.0] learn_light True local_rank 0 multi_gpu False

DatasetNERF: 28 images with shape [1264, 1264] DatasetNERF: 28 images with shape [1264, 1264] Encoder output: 32 dims Using /home/xx/.cache/torch_extensions/py38_cu113 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /home/xx/.cache/torch_extensions/py38_cu113/renderutils_plugin/build.ninja... Building extension module renderutils_plugin... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module renderutils_plugin... iter= 0, img_loss=1.166577, reg_loss=0.334082, lr=0.02999, time=468.3 ms, rem=15.61 m ... iter= 1990, img_loss=0.936701, reg_loss=0.014961, lr=0.01199, time=484.8 ms, rem=4.85 s iter= 2000, img_loss=1.109134, reg_loss=0.014806, lr=0.01194, time=503.9 ms, rem=0.00 s Running validation MSE, PSNR 0.25911513, 6.480

^C^CKilled

strace log: sched_yield() = 0 sched_yield() = 0 sched_yield() = 0 ... sched_yield() = 0 sched_yield() = 0

Thanks for your contribution!

Regards

[jmunkberg]

After the first pass, we run xatlas to create a UV parameterization on the triangle mesh. If the first pass failed to create a reasonable mesh, this step can take quite some time or even fail. How does the mesh look in your case at the end of the first pass?

For memory consumption, you can log the usage using nvidia-smi --query-gpu=memory.used --format=csv -lms 100 when you run to get a feel for the usage. Memory is a function of image resolution, batch size and if you have depth peeling enabled. We ran the results in the paper using HPUs with 32+GB of memory, but it should run on lower-spec GPUs if you decrease the rendering resolution and/or batch size.

[hzhshok]

Thanks @jmunkberg!

Why i raised this issue is because the general command 'nvidia-smi' without parameters showed it only used 7.4G totally, and that GPU memory has 11G, so i suspected that is not memory issue.

Sorry, once i checked the related output including mesh, but i forgot what output it is, and now that host was destroyed, so i am only able to continue to track this after making my new environment.

In addition, the GPU system even was made pending(nvidia-smi was blocked, no any response.) every time train.py failed. I understand why this feature uses Starvation algorithm for the GPU memory allocation, but,

Does team have the plan to optimize this feature for it's memory allocation strategy?

Regards

[jmunkberg]

Hello @hzhshok ,

Looking at the error metrics:

MSE, PSNR 0.25911513, 6.480

That's extremely large errors, so I assume the first pass did not train properly. What do the images look like in the out/nerf_handong folder (or the name of your current experiment)? If the reconstruction succeeded, I would expect a PSNR of 25 dB or higher. If the reconstruction fails, it is very hard to create a uv-parameterization (it is hard to uv map a triangle soup), and xatlas would fail/hang.

I suspect something else is wrong already in the first pass. A few things that can affect quality:

• Is the lighting setup constant in all training data?
• Do you have high quality foreground segmentation masks?
• Are you sure that the poses are correct and that the pose indices and corresponding images match?
• Does the training images contain substantial motion or defocus blur

Also, just to verify, is the example from the readme python train.py --config configs/bob.json working without issues on your setup?

[hzhshok]

Hello, Thanks @jmunkberg! Yes, that something should be wrong in the first pass, and i will check the trained effect of the labeled images using current images as the fundamental.

Is the lighting setup constant in all training data? -- No, some of images has the strong light, because this images is from outdoor. Do you have high quality foreground segmentation masks? --Do you mean the high quality foreground is that the image was labeled on the pure background as the examples(chair or other...)? This time, I just used the wild outdoor images to check if this can be done for non-constant light and relatively pure color in images, and did not label the images, and i will check the effect with the high quality foreground.-) In addition, i just want to check the effect of texturing mesh after using the traditional sfm tool. Are you sure that the poses are correct and that the pose indices and corresponding images match? -- Yes, it should be, the colmap can get result although it did not had good effect. Does the training images contain substantial motion or defocus blur -- it no more blur, that images are resized from high resolution 3003x4000 to 1264x1264, i think i should have impact, but

Regards

[hzhshok]

Hello, Now i found other samples about building to try this feature, and labeled the samples to eliminate the background, it seems like passing the first train/mesh , but it failed to do second train/mesh, so please give a suggestion, thanks!

The question: a. Why it breaked off during running? the log seeing the console output? Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: Function texture2d_mipBackward returned an invalid gradient at index 0 - got [1, 4, 4, 3] but expected shape compatible with [1, 5, 5, 3] Maybe the MSE/PSNR are not the good one, if so, could you help give me suggestion please about how to imporve training images aspects? or trainning images not suitable the rule？ b. Why the result have serious blur? see the intermediate picture and two original image sampels. c. How to imporve the last result? input trining images or other.

Hardware: 3080(24G) - win11

Samples: (see the attached two picture for the example) a. 50 images. b. Resolution: 2456(width)x1638(hight) c. Transform parameters with: --aabb_scale 2 for colmap.

{ "ref_mesh": "data/nerf_synthetic/building", "random_textures": true, "iter": 8000, "save_interval": 100, "texture_res": [5120,5120], "train_res": [1638, 2456], "batch": 1, "learning_rate": [0.03, 0.0001], "ks_min" : [0, 0.08, 0.0], "dmtet_grid" : 128, "mesh_scale" : 5, "laplace_scale" : 3000, "display": [{"latlong" : true}, {"bsdf" : "kd"}, {"bsdf" : "ks"}, {"bsdf" : "normal"}], "layers" : 4, "background" : "white", "out_dir": "nerf_building" }

The console log: iter= 8000, img_loss=0.061592, reg_loss=0.016066, lr=0.00075, time=499.0 ms, rem=0.00 s Running validation MSE, PSNR 0.02140407, 17.038 Base mesh has 214359 triangles and 105082 vertices. Writing mesh: out/nerf_building\dmtet_mesh/mesh.obj writing 105082 vertices writing 224020 texcoords writing 105082 normals writing 214359 faces Writing material: out/nerf_building\dmtet_mesh/mesh.mtl Done exporting mesh Traceback (most recent call last): File "D:\zhansheng\proj\windows\nvdiffrec\train.py", line 620, in geometry, mat = optimize_mesh(glctx, geometry, base_mesh.material, lgt, dataset_train, dataset_validate, FLAGS, File "D:\zhansheng\proj\windows\nvdiffrec\train.py", line 428, in optimize_mesh total_loss.backward() File "C:\Users\jinshui\anaconda3\envs\dmodel\lib\site-packages\torch_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\jinshui\anaconda3\envs\dmodel\lib\site-packages\torch\autograd_init_.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: Function texture2d_mipBackward returned an invalid gradient at index 0 - got [1, 4, 4, 3] but expected shape compatible with [1, 5, 5, 3]

Regards

[ZirongChan]

@hzhshok I ran into the same error about "texture2d_mipBackward returned an invalid gradient" , in my case, it is "got [1,2,2,3] but expected [1,3,3,3]". Did you solve this issue? or did you find the reason about it .

@jmunkberg thanks for your great work btw, what would you suggest that can cause this issue? bad segmentation or sth. ?

[jmunkberg]

That is an error from nvdiffrast. I would try to use power-of-two resolutions on the textures and training, e.g.,

    "texture_res": [ 1024, 1024 ],
    "train_res": [1024, 1024],

In case the texture2d_mipBackward is not stable for all (non-pow2) resolutions.

[hzhshok]

@ZirongChan, i am not sure if it is caused by memory, you know i used images near 2kX2k which costed more memory for my single 24G GPU, so I used the strategy of spliting image to small blocks as Jmunkberg talked inside other issue, at least it did not happen such error.

Regards