OpenSplat icon indicating copy to clipboard operation
OpenSplat copied to clipboard

Published 20 hours ago • verified publisher icon pierotofy

A way to continue training

Open Garshishka opened this issue 10 months ago • 6 comments

Considering that this program can use CPU and low VRam cards to train, how about adding a way or parameter to continue training from saved splat.ply? Is this even feasible?

Garshishka avatar Apr 21 '24 11:04 Garshishka

I don't see why not.

  1. Modify savePly to store the current step count (in a comment PLY header value, maybe)
  2. Read PLY back into the tensors (reverse of savePly), read step count.
  3. Resume from the previous step count.

For a numerically correct resume, one should also dump the optimizer state but I don't think that would actually matter too much for the end result.

We'd welcome a pull request for this. Interested?

pierotofy avatar Apr 21 '24 15:04 pierotofy

I would if I could :( But cpp and ML are an unknown to me

Garshishka avatar Apr 21 '24 21:04 Garshishka

I think it would be very useful.

stefvfx avatar Apr 28 '24 12:04 stefvfx

+1 for this feature. Currently I can only reasonably train ~3000 iterations before RAM consumption exhausts my resources because of the memory leak on MPS devices. I am hoping that stopping and resuming the training would reset this, allowing me to train for longer.

Itox001 avatar Jul 06 '24 15:07 Itox001

+1. I'm not a C++ guy so I can't help here.

eloquentarduino avatar Sep 20 '24 11:09 eloquentarduino