Steady RAM Usage Increase During Video Inference using video.py

[chain-of-immortals]

Hello,

I’ve been running some tests using the nano_llm.vision.video module with live camera streaming on AGX Orin 64gb model.

with the following parameters, --model Efficient-Large-Model/VILA1.5-13b
--max-images 5
--max-new-tokens 3
--prompt 'do you see a moniter in the frame? reply in binary 0 is no and 1 is yes'

I noticed a steady increase in RAM usage during these tests and wanted to get some clarification on what might be causing this.

Here are the details:

Setup: First, I used a USB camera streaming at 640x480 resolution. Then, I tested with another camera streaming at 4K resolution.I have attached the graph of the ram usage in both the cases.

Observation: In both cases, I observed a continuous climb in RAM usage over time, which persisted throughout the streaming session. Much quicker ramp up in the case of 4k images. I’m wondering if this behavior could be attributed to how frames are handled or any other aspects of the video processing pipeline in the script. Is there any known issue or specific configuration I should be aware of that might help address this?

Also How should i think about the optimal size of the video frames i should be feeding this OpenVila1.5 13b model?

Any insights or suggestions would be greatly appreciated.

Thank you!