RawTherapee
RawTherapee copied to clipboard
Beep6581
std::bad_alloc when batch processing many images using rawtherapee-cli
Batch processing 10,000 raw frames of raw video, using the command line interface executable, with the command:
"C:\Program Files\RawTherapee\dev_5.8-2178-g6cc9537ab_20200619\rawtherapee-cli.exe" -o "D:\Tiff dump" -t -p "E:\Driving and stuff\Samples\18.pp3" -c "E:\Driving and stuff\BMPCC_1_2020-11-10_0736_C0018*.dng"
I got the first 7,008 frames successfully processed, and then got this message in Windows Command Prompt:
terminate called recursively terminate called recursively terminate called after throwing an instance of 'std::bad_alloc' terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc what(): std::bad_alloc
Link to raw files, where the BMPCC_1_2020-11-10_0736_C0018_007007.dng successfully processed, then the error was raised. Also included is my sidechain file. https://drive.google.com/drive/folders/1mJ2n2sL-2SpOTyDOUXkMNpd7bzI7-svX?usp=sharing
Update: Copy pasted files after the issue to new directory for rendering the rest, and the 7009th frame processed just fine. Will update again after the remaining 3,000 frames process.
Rawtherapee version info: Version: 5.8-2178-g6cc9537ab Branch: dev Commit: 6cc9537ab Commit date: 2020-06-19 Compiler: gcc\ 10.1.0 Processor: generic x86 System: Windows Bit depth: 64 bits Gtkmm: V3.24.2 Lensfun: V0.3.2.0 Build type: Release Build flags: -mwin32\ -m64\ -mthreads\ -msse2\ -std=c++11\ -mtune=generic\ -Werror=unused-label\ -Werror=delete-incomplete\ -fno-math-errno\ -Wall\ -Wuninitialized\ -Wcast-qual\ -Wno-deprecated-declarations\ -Wno-unused-result\ -Wunused-macros\ -fopenmp\ -Werror=unknown-pragmas\ -Wno-aggressive-loop-optimizations\ -DNDEBUG\ -O3\ -ftree-vectorize Link flags: -m64\ -mthreads\ -static-libgcc\ -mtune=generic\ -s\ -O3 OpenMP support: ON MMAP support: ON Build OS: Linux 4.4.0-101-generic x86_64 Build date: Fri, 19 Jun 2020 21:07:32 +0000 UTC Build epoch: 1592600852 Build UUID: f41dc068-e4b0-4ab4-90ef-71237ada3513
std::bad_alloc
that usually means RT ran out of memory. Never processed such a large batch. Processing such a large number of files in sequence even small memory leaks may sum up to the point where RT runs out of memory. I will have a look, though it will take a while
Wow, I am surprised that there are possible memory leaks in batch command line processing. I assumed that by not having to ram cache previews like in the GUI, I was getting around that, and could process practically indefinitely, given a directory of raw files, a sidechain processing profile, adequate output hard drive space.
Would doing another huge batch processing test with task manager running and screenshot the results be helpful in verifying that your suspicion is correct?
Update: bug is not consistently repeatable. I processed a 25,500 image batch with nothing else running (not sure if that was the case with the time I had the isue.) and did not run into any issues. I will do more tests with a ram usage tracker.
I seem to consistently be having this same issue also, however i've been using the batch processing queue from the GUI. I first noticed it when running a batch of ~50 nikon .NEF raw files on my laptop with 8GB RAM. it managed to get through ~15 images before it had completely filled my memory and crashed.
I then added an additional 8GB of swap and kept an eye on it. It seems to get through ~30 images before both my swap space & RAM are both full. Notably the RAM usage does not seem to reduce until I completely close rawtherapee, even if I "switch off" batch processing.
I need to do another batch in the coming days, I'll post a video of what's happening. with htop open & log files
please see attached video's showing the memory use climbing and not being free'd until the program terminates https://youtu.be/o0Zr6W8LAbE (memory leak) https://youtu.be/RRhuKE5G2Js (free'd on close) Please note i'm uploading these on slow hotel wifi so they may not be viewable for a while after posting