No transforms.json produced after running demo colab notebook on custom video or images

[SaifeldeenAdel]

Describe the bug I'm running the latest version of the demo.ipynb and I reach the Data Processing cell and I run it, upload my video and wait. It says its done converting my videos to images and the cell says its done running. The output path is corrent as I can see folders "colmap" (which is empty), images, images_2, images_4, images_8 all with my images, and thats it. There is no transforms.json. According to the docs, the ns-process-data command should be producing the transforms.json. Why is it not?

To Reproduce Steps to reproduce the behavior:

1. Go to the demo.ipynb colab notebook in this repository
2. Run first two cells
3. Upload video
4. Wait till the cell is done and check if theres a transforms.json file, there won't be.

Expected behavior I expected to find a transforms.json that has my camera poses so i can start training.

OS: Windows 10 (but im running on google colab)

[AntonioMacaronio]

Hey thanks for the report, we have a bugfix in the making but the issue is with the colmap build - for now it can be fixed if you run this branch but this will be addressed by the end of this week

[SaifeldeenAdel]

Hey thanks for the report, we have a bugfix in the making but the issue is with the colmap build - for now it can be fixed if you run this branch but this will be addressed by the end of this week

Thank you! But sorry I think im missing something, you linked a pull request but you said branch? I'm not super used to dealing with pull requests, is there a way to get to the branch associated with that pull request or?

[AntonioMacaronio]

Yes you're right, my mistake - I put the wrong link but I've updated it to be the branch which contains the fix - if you don't want to pull the new branch cause of potential merge conflicts, you can just copy and paste the new lines into your demo notebook - keep in mind it does take up to 45 minutes to build colmap from scratch with CUDA

[SaifeldeenAdel]

Gotcha, thanks again. That wont be an issue as I was just testing the notebook and wasn't really pulling from the repo so I'll just test using your notebook instead.

it does take up to 45 minutes to build colmap from scratch with CUDA

damn alright, ill look out for that. Thanks!

[kesitrifork]

@AntonioMacaronio Hi, I tried your code and have now colmap with cuda support on Google Colab, however running ns-process-data images still does not produce a transforms.json

Did you succeed with this? it seems like something has broken with ns-process-data, it only copies the files.

[AntonioMacaronio]

Perhaps you can try to run the command in the terminal command line interface (just copy and paste the command into the terminal given by the ns-train cell using the colab-xterm module - this is just an idea for a workaround as colab has always struggled with nerfstudio's rich text output

[kesitrifork]

@AntonioMacaronio I am running in colab-xterm, and verifying in the file browser also what is generated.

It does not seem to work. I think the problem is that Colab is now on Cuda 12.2 and nothing else has updated to that yet, neither Pytorch, Colmap or Nerfstudio and gives a lot of problems.

[noureldinabdelalim]

I am also using colab however I'm facing a slightly different issue where the images are being copied into the folders successfully but then the transforms.json file only has 2 images out of the 60 i uploaded. Any fix for this? Thanks

[noureldinabdelalim]

the cell is terminating after giving me the message "Done copying images with prefix frame", @AntonioMacaronio do you know what could I do to fix this or what might be the cause?

[AntonioMacaronio]

@kesitrifork @noureldinabdelalim @SaifeldeenAdel I was able to get it working with the xterm module. It took about 16 minutes to generate all the camera poses for 300 frames of a video on my end.

There should be no conflicts with Cuda, Pytorch, Colmap, or Nerfstudio; the actual error is that Google Colab handles the rich text output of nerfstudio very poorly, and it always hangs if any nerfstudio script/command prints rich-formatted text to the console. So the fix I found was to use a module to create a makeshift terminal that can handle the cell outputs. This should allow COLMAP to run to completion and generate the full transforms.json file needed for NeRF training.

Here is a (temporary, but it will always work) fix:

1. Create a jupyter cell like the following:

2. Execute this cell. When you run this cell, a terminal window should pop up beneath.

3. In the terminal window, you need to type in the ns-process-data command, and specify your media type, data directory, and output directory. For me, I was processing a video called "soda-nerf-small.mov" which I manually dragged and dropped into the google colab file explorer, and I specified my input and output folders like so: ns-process-data video --data /content/data/nerfstudio/custom_data/soda-nerf-small.mov --output-dir /content/data/nerfstudio/custom_data/

4. Execute the command and wait. This can take very long for long videos, as COLMAP is running Structure-from-Motion to extract the camera poses. For 300 frames, this took me about 16 minutes.

I will start working on a new PR to address this but yeah it's unlucky that colab in general is always so finnicky.

[noureldinabdelalim]

No matter the number of images or frames entered, it only matches two images, this could have something to do with colmap, is there anyway to overcome it?

[AntonioMacaronio]

@noureldinabdelalim Did you try downloading the COLMAP binary from this PR? You can just copy and paste the first cell there because it installs COLMAP with CUDA which can help with the processing

[noureldinabdelalim]

@AntonioMacaronio The problem still persists, I have been told it is related to colmap 3.9 and that colmap 3.8 would work much better and would give me better results that just the 2 images i keep receiving over and over. When I tried installing colmap 3.8, a huge amount of errors appeared in the installation process so I was unable to do so. Do you have any suggestion as to what i can do to have more than just 2 images match out of 150?

[liuzongyue6]

@AntonioMacaronio Have you encountered this issue that colmap does not work in the colab xterm mode? I am struggling to make the nerfstudio to run on the custom data as no transforms.json created

/content# ns-process-data images --data /content/data/nerfstudio/custom_data/raw_images --output-dir /content/data/nerfstudio/custom_data/images [18:36:03] 🎉 Done copying images with prefix 'frame_'. process_data_utils.py:340 ────────────────────────────────────────────── 💀 💀 💀 ERROR 💀 💀 💀 ─────────────────────────────────────────────── Error running command: colmap feature_extractor --database_path /content/data/nerfstudio/custom_data/images/colmap/database.db --image_path /content/data/nerfstudio/custom_data/images/images --ImageReader.single_camera 1 --ImageReader.camera_model OPENCV --SiftExtraction.use_gpu 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── QObject::moveToThread: Current thread (0x5a1256d13a00) is not the object's thread (0x5a1256d17a50). Cannot move to target thread (0x5a1256d13a00)

[zeitsmatt]

I just ran the latest pull and it finished without error--the output has been pushed outside of my terminal history because of three bad runs of ns-train--but there's not transforms.json being outputted by the ns-process-data as reported above. Again, no errors this time.

[AntonioMacaronio]

@noureldinabdelalim After talking with others it could be an issue of COLMAP 3.8, but I don't think it is likely. I'm just not sure perhaps the data you provided is just too difficult (Varying images, exposure, blur, etc.)? It is really hard to replicate this issue on my end

[AntonioMacaronio]

Ok I think this is definitely a COLMAP != 3.8 issue; I am in the works of editing the current PR to address that issue!