Slow traning speed with A100

[DEVECLOVER]

Configure the environment according to the tutorial, test the training example-dtu-scan65 , and find that the training speed is very slow, however the tutorial says that the 3090 can reach the training speed of 15 minutes and 20K iterations:

my training command 、info and progress as follows:

ns-train neus-facto --pipeline.model.sdf-field.inside-outside False --vis viewer --experiment-name neus-facto-dtu65 sdfstudio-data --data data/sdfstudio-demo-data/dtu-scan65

As we can see, the GPU utilization is very low, I haven't made any changes to the source code, don't know what caused it. Criticism and correction are welcome, thank you!!!

[niujinshuchong]

Hi, this is strange. In our cluster, we found the code will try to use all available cpus and there might be some conflicts in data loader so it's much slower. You could try to add OMP_NUM_THREADS=4 to the training command. This solve our issue but I am not sure if it is helpful for you.

[DEVECLOVER]

@niujinshuchong Thanks a lot. By adding OMP_NUM_THREADS=4, the training speed is indeed improved, but the GPU-Util is only about 40%, please is there any other way to further improve the GPU-Util? Thanks!!!