nerfstudio icon indicating copy to clipboard operation
nerfstudio copied to clipboard

Adding Terminal Window to Jupyter Cells so all Colab users (free and pro) can train with Nerfstudio

Open AntonioMacaronio opened this issue 1 year ago • 3 comments

Background and Issues:

  • All colab users (free and pro) were not be able to run ns-train in a jupyter cell, and this has been made apparent in several issues below. (Pro users can get around this, but it's still problematic and explained below)
  • https://github.com/nerfstudio-project/nerfstudio/issues/2283
  • https://github.com/nerfstudio-project/nerfstudio/issues/1836
  • https://github.com/nerfstudio-project/nerfstudio/issues/1832
  • https://github.com/nerfstudio-project/nerfstudio/issues/1113
  • https://github.com/nerfstudio-project/nerfstudio/issues/2166

The culprit of this bug is that as of Feb 2024, Google Colab runs python 3.10, which has an issue with nerfstudio's rich console outputs. In particular, a python module called shutil will set the terminal size to 0 when dealing with rich console outputs. As a result, nothing gets printed, and Colab will shutdown the cell and terminate early before training can begin.

This is mentioned in other python modules which use the rich console output, and a complete breakdown on the issue is detailed here.

Overview of Changes

  • By using the colab-xterm module, we are able to create a terminal window in a jupyter cell output.
  • After installing it, we clear the cell output to make it very user-friendly to follow the instructions.
  • The command is printed and instructions are given to help the user run ns-train and they will be able to generate the viser/viewer link to see the training progress
  • Removed the "Connecting to Viewer Cell" as users can simply use ns-train's sharable URL link
  • Final change was to add the torchaudio dependency which was mentioned here so that users can train.

Impact and Other Notes

  • While Pro Colab users can get around this by running ns-train in their terminal window, Colab only allows for 1 terminal window. Therefore, to run the render command, pro users need to Ctrl+C the training process hosting the viser/viewer link, which can be quite irritating - especially if they want to make more than just 1 render path.

  • Therefore, even once Colab upgrades to python 3.11 in April 2024, there is still a need to have at least 2 terminal windows, which is possible with this change.

  • To run ns-train, users (free and pro) simply need to follow the instructions to copy and paste the command into the terminal window. This is pretty clear to follow, and looks like the following: image

  • Below is what the cell terminal window looks like after copy+pasting the command on a free colab account: image

AntonioMacaronio avatar Feb 04 '24 11:02 AntonioMacaronio

Just found this as a fix to my own issue with getting the colab notebook to work. This does look to be working (I'm currently at 5% trained on one of the demo scenes), but I did notice you had a little error in the imports. There is now an extra " on the end of an import line. Just wanted to point this out and the non-blocking error it causes before you try to merge this

Screenshot 2024-02-05 at 11 23 24 AM

jfenton888 avatar Feb 05 '24 16:02 jfenton888

@jfenton888 thank you for the catch! I've updated the PR with this change

AntonioMacaronio avatar Feb 05 '24 19:02 AntonioMacaronio

Thank you!!! Going to test this soon and see if anything is needed, but it looks like everything is good!

ginazhouhuiwu avatar Feb 06 '24 02:02 ginazhouhuiwu