BundleTrack icon indicating copy to clipboard operation
BundleTrack copied to clipboard

gpu memory not realesed

Open royzhang12 opened this issue 2 years ago • 4 comments

Hi @wenbowen123, thank you for your nice work. I've been trying to run Bundletrack on a collections of video clips. To automatically swith between each clips, I simply declare Bundler by *Bundler bundler = nullptr; before the main loop of video and construct it again after swith to to a new video clips by delete bundler; bundler = new Bundler(yml,&data_loader); . The pose estimation result seem well. However, I find that the GPU memory is not realesed after swith to a new video clips and the usage keeps increaseting. Untill 10G GPU memory was used, the programe will be automatically killed by no reason.

Do you have any idea of how to resolve this issue?

royzhang12 avatar Oct 11 '22 07:10 royzhang12

Hi, there might be a memory leak. I'll try to find a time to investigate further after a deadline recently. If you are running for some offline data, perhaps you can run each video sequence by bash script? Each time modify the data_dir in the config file to run.

wenbowen123 avatar Oct 11 '22 22:10 wenbowen123

Hi, there might be a memory leak. I'll try to find a time to investigate further after a deadline recently. If you are running for some offline data, perhaps you can run each video sequence by bash script? Each time modify the data_dir in the config file to run.

Hi, @wenbowen123. Thank you for the swift reply. I am doing test on both online and offline data. The suggestion is a great solution for offline data, thanks. However, for oneline test, it does not work. I tried to restart the process autoamtically online, which consumes much time and leads to really slow inference speed. Hope you can help to find the reason cause the memory leak after your deadline. Many thanks.

royzhang12 avatar Oct 12 '22 02:10 royzhang12

I also have this problem right now. I tried the workaround to split my dataset into smaller segments, but it would be great if there's a solid solution.

ChrisSun99 avatar Nov 24 '22 02:11 ChrisSun99

FYI, we recently released a follow up version BundleSDF, which should not have such issue.

wenbowen123 avatar Jul 31 '23 18:07 wenbowen123