NVlabs
Clearer progress tracking
I think clearer progress tracking would be really helpful. With the current logging output it's hard to understand how long training will still take. The output is quite verbose, even at the least verbose debugging level (0), and the interplay of parallel optimization processes makes it even harder to keep an overview. I get that debugging output has to be verbose in order to be able to see all the components working, but I think the problem is to cram everything into one console log to begin with.
Ideally we would have something like a weights & biases (wandb) integration for tracking different progress metrics (e.g. percentage of frames already processed, losses, etc.) on top of the console log. I might look into this but no guarantees. Also, I think there are console logging libraries that fix some (e.g. global) progress bars in place so that one can keep an overview. This might also help.
Kudos to the authors though for this great project. I am only criticizing because I think this project is awesome and making it more accessible would let it come to its full potential! Thanks a lot for the project!
Currently from GUI you can see the current processing frame ID. From there you can infer the progress. But you brought up a really good point. We may add that into the roadmap.
