autonomousvision
Bakedangelo Training
Hello,
Gonna start by saying I am not a very technical person but able to get things to work (somehow!)
I am using bakedangelo on a custom dataset on my 4090 gpu. Very large dataset, ~1200 images. It has been running for about 2 weeks and says it is ~16% complete. I believe it is training for 1 million iterations. I am wondering
- Does this sound right?
- When setting it up, i had to use 86 vs 89 for tinycudann because cuda 11.3 doesnt have support for 89 (which my 4090 is?). Would i be able to use cuda 11.8 to setup tinycudann with 89 and then switch to 11.3 for sdfstudio to work?
- I can't see anything in the viewer, that's correct, correct?
- Does it save checkpoints if i didn't specify it to save checkpoints to view? Where would the be saved at if yes?
- Can I/how do I stop this early and save a checkpoint to continue training at a later date/overnight? (Have been on vacation for these two weeks so not been using the computer, hoping it would have been finished by the time i got back!)
And and all help is appreciated! Screenshot of remote desktop showing the training.
@cgallik This is very strange. It is very slow on your machine. Mine is around 130ms per iteration. I think we could use --vis wandb or --vis tensorboard for monitoring the training since using viewer is slow for backedangelo (not sure if this is the reason why it's slow in your case). You could check whether there is some checkpoints saved in the output/log folder.
there is a checkpoint saved in the outputs folder, saves every 20k. how would i use this? do i opoen up another anaconda prompt and try and run it from the checkpoint with the wandb viewer?
I know I am replying to an old thread, but my results were roughly 20 sec per itt on a 3090, so I am confused as to how @niujinshuchong had their performance be so superior. I am also using cuda_11.8 per nvecc --version.
@ThomasWarn The reasonable speed is less then 200ms per iteration on 3090 if I remember correctly. Could you please try to train other models to check how it's the speed such as bakedsdf?
