NeuralRecon-W icon indicating copy to clipboard operation
NeuralRecon-W copied to clipboard

Published 20 hours ago • verified publisher icon zju3dv

Image downscale failure in data generation

Open ErinH47 opened this issue 2 years ago • 6 comments

Thanks a lot for this excellent project! I encountered a problem when conducting data setup with image downscale parameter > 1. I guess there may be a bug in datasets/phototourism.py line 576 where the function get_colmap_depth() returns zero outputs (both depth & weight).

my data generation setup is as follows: (data_generation.sh)

#!/bin/bash set -x set -u set -e

now=$(date +"%Y%m%d_%H%M%S") echo "working directory is $(pwd)" jobname="data-generation-$1-$now" export CUDA_VISIBLE_DEVICES=4 dataset_name="phototourism" cache_dir="cache_sgs" root_dir=$1 min_observation=-1

if [ ! -f $root_dir/*.tsv ]; then python tools/prepare_data/prepare_data_split.py
--root_dir $root_dir
--num_test 10
--min_observation $min_observation --roi_threshold 0 --static_threshold 0 fi python tools/prepare_data/prepare_data_cache.py
--root_dir $root_dir
--dataset_name $dataset_name --cache_dir $cache_dir
--img_downscale 8
--semantic_map_path semantic_maps --split_to_chunks 64
2>&1|tee log/${jobname}.log

my output log:

Preparing cache for scale 8... reading sfm result from sparse... Note: training near far will generate from sparse voxel!!!! Reading images.bin.. Reading cameras.bin.. Compute c2w poses.. Generating rays and rgbs..

0%| | 0/100040 [00:00<?, ?it/s] 11%|█▏ | 11423/100040 [00:00<00:00, 114219.22it/s] 23%|██▎ | 23138/100040 [00:00<00:00, 115935.24it/s] 35%|███▍ | 34792/100040 [00:00<00:00, 116205.60it/s] 46%|████▋ | 46515/100040 [00:00<00:00, 116608.39it/s] 58%|█████▊ | 58176/100040 [00:00<00:00, 116323.60it/s] 70%|██████▉ | 69809/100040 [00:00<00:00, 115938.68it/s] 82%|████████▏ | 81551/100040 [00:00<00:00, 116420.19it/s] 93%|█████████▎| 93248/100040 [00:00<00:00, 116593.45it/s] 100%|██████████| 100040/100040 [00:00<00:00, 116292.81it/s] 2022-08-01 19:07:19.068 | DEBUG | tools.prepare_data.generate_voxel:gen_octree_from_sfm:59 - Points filtered from raw point cloud: 27595/100040 2022-08-01 19:07:19.782 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:124 - number of points for voxel generation: 725832/744660 2022-08-01 19:07:22.507 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:147 - level: 5 for expected voxel size: 0.25 2022-08-01 19:07:24.106 | DEBUG | tools.prepare_data.generate_voxel:gen_octree_from_sfm:59 - Points filtered from raw point cloud: 27595/100040 2022-08-01 19:07:46.472 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:124 - number of points for voxel generation: 3862151/3893596 2022-08-01 19:07:46.507 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:147 - level: 5 for expected voxel size: 0.25 Mean Projection Error: tensor(0.8717)

0%| | 0/1353 [00:00<?, ?it/s]

0%| | 0/1 [00:00<?, ?it/s][A 100%|██████████| 1/1 [00:00<00:00, 76.60it/s]

0%| | 0/1 [00:00<?, ?it/s][A 100%|██████████| 1/1 [00:00<00:00, 98.90it/s]

0%| | 0/1353 [00:01<?, ?it/s] padding valid depth percentage: from 0.0 to 0.2 with padding 1592 Traceback (most recent call last): File "tools/prepare_data/prepare_data_cache.py", line 176, in dataset = dataset_dictargs.dataset_name File "/DATA/disk1/NeuralRecon-W/./datasets/phototourism.py", line 122, in init self.read_meta() File "/DATA/disk1/NeuralRecon-W/./datasets/phototourism.py", line 671, in read_meta paddings_rays = rays[valid_depth, :][pad_ind] IndexError: index is out of bounds for dimension with size 0

ErinH47 avatar Aug 01 '22 13:08 ErinH47

Hi, thanks for your interest in our work! The problem seems to be there is no SfM key point exists in that image. The latest code has fixed this issue.

Burningdust21 avatar Aug 02 '22 01:08 Burningdust21

@Burningdust21 Thanks for your help! And I have another problem, CUDA often ran out of memory when conducting sdf updating to octree. Is there a hyperparam that I can adjust? Thanks.

ErinH47 avatar Aug 02 '22 13:08 ErinH47

Hi, there are two things you can do to reduce gpu memory usage.

First, set a smaller batch size in here.

Second, enlarge voxel size in config, this can reduce the size of octree and therefore reduce gpu memory. We found that adjusting voxel size to make octree level to be 10 is a good trade off between accuracy and gpu memory. Octree level is printed in the beginning of training with the format "trining octree will be in level: 10, with respect to voxel size 0.0636774228515625".

Burningdust21 avatar Aug 09 '22 03:08 Burningdust21

Hi , I have got the same problem with the newest code. How to fix it? thanks!

the log is the following:

(nerf) mine@WS-E900-G4:~/dev/NeuralRecon-W$ ./scripts/data_generation.sh data/palacio_de_bellas_artes

  • set -u
  • set -e ++ date +%Y%m%d_%H%M%S
  • now=20230209_211441 ++ pwd
  • echo 'working directory is /home/cqiu/dev/NeuralRecon-W' working directory is /home/cqiu/dev/NeuralRecon-W
  • jobname=data-generation-data/palacio_de_bellas_artes-20230209_211441
  • export CUDA_VISIBLE_DEVICES=0
  • CUDA_VISIBLE_DEVICES=0
  • dataset_name=phototourism
  • cache_dir=cache_sgs
  • root_dir=data/palacio_de_bellas_artes
  • min_observation=-1
  • '[' '!' -f data/palacio_de_bellas_artes/palacio_de_bellas_artes.tsv ']'
  • python tools/prepare_data/prepare_data_cache.py --root_dir data/palacio_de_bellas_artes --dataset_name phototourism --cache_dir cache_sgs --img_downscale 1 --semantic_map_path semantic_maps --split_to_chunks 64
  • tee log/data-generation-data/palacio_de_bellas_artes-20230209_211441.log tee: log/data-generation-data/palacio_de_bellas_artes-20230209_211441.log: No such file or directory Preparing cache for scale 1... reading sfm result from ../neuralsfm... Note: training near far will generate from sparse voxel!!!! Reading images.bin.. Reading cameras.bin.. Compute c2w poses.. Generating rays and rgbs.. 100%|██████████| 1668354/1668354 [00:18<00:00, 91957.82it/s] 2023-02-09 21:18:07.080 | DEBUG | tools.prepare_data.generate_voxel:gen_octree_from_sfm:59 - Points filtered from raw point cloud: 61070/367331 2023-02-09 21:18:08.881 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:124 - number of points for voxel generation: 1427637/1648512 2023-02-09 21:18:10.113 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:147 - level: 5 for expected voxel size: 0.1 2023-02-09 21:18:14.219 | DEBUG | tools.prepare_data.generate_voxel:gen_octree_from_sfm:59 - Points filtered from raw point cloud: 61070/367331 2023-02-09 21:19:10.923 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:124 - number of points for voxel generation: 7912564/8567904 2023-02-09 21:19:10.977 | DEBUG | tools.prepare_data.generate_voxel:gen_octree:147 - level: 6 for expected voxel size: 0.1 Mean Projection Error: tensor(1.1654) 100%|██████████| 40/40 [00:00<00:00, 310.42it/s] [WARNING] batch has 0 intersections!!306.27it/s] [WARNING] batch has 0 intersections!! 100%|██████████| 40/40 [00:00<00:00, 192.87it/s] 0%| | 1/1624 [00:02<1:05:37, 2.43s/it][WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.003795979618484957 to 0.4 with padding 1660253 sample depth percent after padding: 0.380859375 100%|██████████| 30/30 [00:00<00:00, 325.61it/s] 100%|██████████| 30/30 [00:00<00:00, 192.04it/s] 0%| | 2/1624 [00:03<39:03, 1.44s/it] padding valid depth percentage: from 0.0035264146338156127 to 0.4 with padding 1559400 sample depth percent after padding: 0.4130859375 100%|██████████| 15/15 [00:00<00:00, 493.32it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 15/15 [00:00<00:00, 338.16it/s] 0%| | 3/1624 [00:03<24:00, 1.12it/s][WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.010713243873978996 to 0.4 with padding 266895 sample depth percent after padding: 0.4052734375 100%|██████████| 30/30 [00:00<00:00, 582.13it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 30/30 [00:00<00:00, 425.65it/s] 0%| | 4/1624 [00:03<18:52, 1.43it/s][WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.00451176661647031 to 0.4 with padding 334850 sample depth percent after padding: 0.3916015625 100%|██████████| 7/7 [00:00<00:00, 327.65it/s] 100%|██████████| 7/7 [00:00<00:00, 217.55it/s] 0%| | 5/1624 [00:03<13:29, 2.00it/s]padding valid depth percentage: from 0.019568072585702236 to 0.4 with padding 310416 sample depth percent after padding: 0.4169921875 100%|██████████| 27/27 [00:00<00:00, 479.19it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 27/27 [00:00<00:00, 309.59it/s] 0%| | 6/1624 [00:04<12:45, 2.11it/s][WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.0052823132736603854 to 0.4 with padding 565042 sample depth percent after padding: 0.4033203125 100%|██████████| 7/7 [00:00<00:00, 363.73it/s] 100%|██████████| 7/7 [00:00<00:00, 230.53it/s] 0%| | 7/1624 [00:04<09:47, 2.75it/s]padding valid depth percentage: from 0.012324164184225288 to 0.4 with padding 237288 sample depth percent after padding: 0.390625 100%|██████████| 27/27 [00:00<00:00, 270.20it/s] 100%|██████████| 27/27 [00:00<00:00, 149.95it/s] 0%| | 8/1624 [00:05<13:22, 2.01it/s]padding valid depth percentage: from 0.0032658063557249174 to 0.4 with padding 1663889 sample depth percent after padding: 0.3759765625 100%|██████████| 26/26 [00:00<00:00, 345.53it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 26/26 [00:00<00:00, 199.35it/s] 1%| | 9/1624 [00:05<14:03, 1.92it/s]padding valid depth percentage: from 0.003747473907180131 to 0.4 with padding 1167002 sample depth percent after padding: 0.4072265625 100%|██████████| 22/22 [00:00<00:00, 522.21it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 22/22 [00:00<00:00, 396.83it/s] 1%| | 10/1624 [00:06<12:08, 2.22it/s][WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.005304033782789344 to 0.4 with padding 232421 sample depth percent after padding: 0.4345703125 100%|██████████| 11/11 [00:00<00:00, 443.60it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! 100%|██████████| 11/11 [00:00<00:00, 290.21it/s] 1%| | 11/1624 [00:06<09:51, 2.73it/s][WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.018117994997166903 to 0.4 with padding 276326 sample depth percent after padding: 0.4013671875 100%|██████████| 30/30 [00:00<00:00, 491.79it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 30/30 [00:00<00:00, 292.68it/s] 1%| | 12/1624 [00:06<10:57, 2.45it/s][WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.003304916895170812 to 0.4 with padding 685982 sample depth percent after padding: 0.392578125 100%|██████████| 30/30 [00:00<00:00, 316.83it/s] 100%|██████████| 30/30 [00:00<00:00, 189.53it/s] 1%| | 13/1624 [00:07<13:59, 1.92it/s]padding valid depth percentage: from 0.004232961372431143 to 0.4 with padding 1652395 sample depth percent after padding: 0.40234375 100%|██████████| 26/26 [00:00<00:00, 308.53it/s] [WARNING] batch has 0 intersections!!] 100%|██████████| 26/26 [00:00<00:00, 181.92it/s] 1%| | 14/1624 [00:08<15:10, 1.77it/s]padding valid depth percentage: from 0.004291407527001244 to 0.4 with padding 1447690 sample depth percent after padding: 0.3916015625 100%|██████████| 31/31 [00:00<00:00, 414.22it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 31/31 [00:00<00:00, 271.07it/s] 1%| | 15/1624 [00:08<15:16, 1.76it/s][WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.00354900870465379 to 0.4 with padding 944674 sample depth percent after padding: 0.3974609375 100%|██████████| 30/30 [00:00<00:00, 347.59it/s] 100%|██████████| 30/30 [00:00<00:00, 223.50it/s] 1%| | 16/1624 [00:09<15:56, 1.68it/s]padding valid depth percentage: from 0.002409877623313861 to 0.4 with padding 1175508 sample depth percent after padding: 0.3974609375 100%|██████████| 9/9 [00:00<00:00, 431.20it/s] [WARNING] batch has 0 intersections!! 100%|██████████| 9/9 [00:00<00:00, 284.03it/s] 1%| | 17/1624 [00:09<12:27, 2.15it/s]padding valid depth percentage: from 0.01747449398269858 to 0.4 with padding 213652 sample depth percent after padding: 0.4072265625 100%|██████████| 5/5 [00:00<00:00, 601.96it/s] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 5/5 [00:00<00:00, 499.83it/s] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.03199779800099778 to 0.4 with padding 35653 sample depth percent after padding: 0.4091796875 100%|██████████| 30/30 [00:00<00:00, 355.61it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! 100%|██████████| 30/30 [00:00<00:00, 208.06it/s] 1%| | 19/1624 [00:10<11:20, 2.36it/s]padding valid depth percentage: from 0.0032718840951455152 to 0.4 with padding 1361074 sample depth percent after padding: 0.41796875 100%|██████████| 8/8 [00:00<00:00, 290.62it/s] 100%|██████████| 8/8 [00:00<00:00, 190.36it/s] 1%| | 20/1624 [00:10<09:52, 2.71it/s]padding valid depth percentage: from 0.009606116503133644 to 0.4 with padding 477251 sample depth percent after padding: 0.390625 100%|██████████| 27/27 [00:00<00:00, 395.46it/s] 100%|██████████| 27/27 [00:00<00:00, 273.72it/s] 1%|▏ | 21/1624 [00:11<10:46, 2.48it/s]padding valid depth percentage: from 0.005211155744026883 to 0.4 with padding 917940 sample depth percent after padding: 0.3798828125 100%|██████████| 30/30 [00:00<00:00, 467.45it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! 100%|██████████| 30/30 [00:00<00:00, 341.70it/s] 1%|▏ | 22/1624 [00:11<11:14, 2.38it/s][WARNING] batch has 0 intersections!! [WARNING] batch has 0 intersections!! padding valid depth percentage: from 0.005267733729596029 to 0.4 with padding 551640 sample depth percent after padding: 0.396484375 100%|██████████| 8/8 [00:00<00:00, 287.82it/s] 100%|██████████| 8/8 [00:00<00:00, 174.84it/s] 1%|▏ | 23/1624 [00:11<09:36, 2.78it/s]padding valid depth percentage: from 0.007030822972534287 to 0.4 with padding 472105 sample depth percent after padding: 0.4013671875 100%|██████████| 27/27 [00:00<00:00, 338.67it/s] [WARNING] batch has 0 intersections!!] [WARNING] batch has 0 intersections!! 100%|██████████| 27/27 [00:00<00:00, 215.08it/s] 1%|▏ | 24/1624 [00:12<11:34, 2.30it/s]padding valid depth percentage: from 0.0039525877907908005 to 0.4 with padding 980787 sample depth percent after padding: 0.3876953125 100%|██████████| 1/1 [00:00<00:00, 417.43it/s] 100%|██████████| 1/1 [00:00<00:00, 444.45it/s] 1%|▏ | 24/1624 [00:12<13:51, 1.92it/s] padding valid depth percentage: from 0.0 to 0.4 with padding 101 Traceback (most recent call last): File "/home/cqiu/dev/NeuralRecon-W/tools/prepare_data/prepare_data_cache.py", line 176, in dataset = dataset_dictargs.dataset_name File "/home/cqiu/dev/NeuralRecon-W/./datasets/phototourism.py", line 123, in init self.read_meta() File "/home/cqiu/dev/NeuralRecon-W/./datasets/phototourism.py", line 671, in read_meta paddings_rays = rays[valid_depth, :][pad_ind] IndexError: index is out of bounds for dimension with size 0

chufall avatar Feb 09 '23 13:02 chufall

I also meet this problem. I just add some code in phototourism.py to solve this problem, but it seems not to work. image

czg-first avatar Mar 21 '23 13:03 czg-first

Hi, Have you solved this issue ? I have the same situation when image_downscale is set to 8

lt-xiang avatar Jul 28 '23 16:07 lt-xiang