vadimkantorov
std::bad_alloc
Hi. I tried to run mpegflow on Ubuntu 14.04 (clean) with *.mp4 file (h264 video) as in example: ./mpegflow filename.mp4 > filename.txt But it consumed all available RAM (72 GB) and then showed bad_alloc error. Did I do something wrong? Using of mpegflow with other arguments gave the same results.
Hi, It seems like a bug (and possibly FFmpeg's bug). Unfortunately, I won't have time to look at it till March 14 (ECCV deadline). Let me know if you could find the cause before, and if not, send me the file, I could take a look after March 14.
Hi @vadimkantorov I find that in mpegflow.cpp line 303 there is a static variable prev, since it is a vector, so pre.clear() cannot release its memory, as discussed in: http://stackoverflow.com/questions/13944886/is-stdvector-memory-freed-upon-a-clear so with a long video, prev eats more and more memory, till bad_alloc happens.
I wonder wether we can use two variables prev1 and prev2 to replace vector in line 356? I hope there are only two prevs we need so we can solve the problem.
Hello, Has this issue been resolved by anyone ? There doesn't seem to be any problem with running the code on mpeg4 video files. After trsting the code on a large dataset of h264 videos, it crashs giving the bad_alloc error regardless of the length of video, however short.
@ZhichenZhao I'll take a look this wekend to avoid doing clear on a statically allocated vector.
@Al-Bas Could you identify which line / code piece causes the crash? Is it the same issue @ghost has reported? Can you send me to [email protected] your video and ffmpeg libraries / includes?
Many thanks - Sent an email containing all the details. Please let me know if you'd like me to run any tests on my machine or send extra info etc --
The following is the back trace log:
#0 0x00007ffff6e02cc9 in __GI_raise (sig=sig@entry=0x6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007ffff6e060d8 in __GI_abort () at abort.c:89 #2 0x00007ffff74f76b5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007ffff74f5836 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007ffff74f5863 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007ffff74f5aa2 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007ffff74f5f8d in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x0000000000458279 in allocate (this=0x11f0500 <output_vectors_std(int, long, char, std::vector<AVMotionVector, std::allocator<AVMotionVector> >&)::prev>, __n=
) at /usr/include/c++/4.8/ext/new_allocator.h:104 #8 _M_allocate (this=0x11f0500 <output_vectors_std(int, long, char, std::vector<AVMotionVector, std::allocator<AVMotionVector> >&)::prev>, __n= ) at /usr/include/c++/4.8/bits/stl_vector.h:168 #9 std::vector<FrameInfo, std::allocator<FrameInfo> >::_M_insert_aux (this=this@entry=0x11f0500 <output_vectors_std(int, long, char, std::vector<AVMotionVector, std::allocator<AVMotionVector> >&)::prev>, __position=__position@entry=..., __x=...) at /usr/include/c++/4.8/bits/vector.tcc:345 #10 0x000000000045728d in push_back (__x=..., this=0x11f0500 <output_vectors_std(int, long, char, std::vector<AVMotionVector, std::allocator<AVMotionVector> >&)::prev>) at /usr/include/c++/4.8/bits/stl_vector.h:913 #11 output_vectors_std (frameIndex=frameIndex@entry=0x3, pts=0x7d2, pictType= , motionVectors=...) at mpegflow.cpp:319 #12 0x0000000000456119 in main (argc=argc@entry=0x2, argv=argv@entry=0x7fffffffdfd8) at mpegflow.cpp:418 #13 0x00007ffff6dedec5 in __libc_start_main (main=0x455fb0 <main(int, char const**)>, argc=0x2, argv=0x7fffffffdfd8, init= , fini= , rtld_fini= , stack_end=0x7fffffffdfc8) at libc-start.c:287 #14 0x00000000004561c9 in _start ()
So it is crashing when calling output_vectors_std -- Apologies about the confusion earlier..
@Al-Bas Hi,
Thanks, I received your e-mail with video. I'll take a look at it tomorrow.
Meanwhile, does the problem go away if you replace the line https://github.com/vadimkantorov/mpegflow/blob/master/mpegflow.cpp#L366 to say vector<AVMotionVector>().swap(prev); as @ZhichenZhao suggested?
@Al-Bas also, could you put a fprintf(stderr, "dummy_pts: %lld, pts: %lld\n", dummy_pts, pts); just before https://github.com/vadimkantorov/mpegflow/blob/master/mpegflow.cpp#L312? What does the output look like?
Basically in that piece of code I'm adding dummy frames for non-existent PTS. There could be a problem if timestamps in the video frames differ by gigantic amounts.
Hello @vadimkantorov,
Basically in that piece of code I'm adding dummy frames for non-existent PTS. There could be a problem if timestamps in the video frames differ by gigantic amounts.
That's true enough ! I've been testing it on a bunch of h264 .mp4 videos from the recent Sports video dataset compiled by Andrej Karpathy found in: https://github.com/gtoderici/sports-1m-dataset/
The program crashed randomly for most videos, but seems to work for a very modest few.
For one video that crashes, using fprintf(stderr, "dummy_pts: %ld, pts: %ld\n", dummy_pts, pts); yields:
dummy_pts: 1, pts: 2002 dummy_pts: 2, pts: 2002 ...... ...... dummy_pts: 1023, pts: 2002 dummy_pts: 1024, pts: 2002
I have edited the code to skip appending dummies to the prev vector and it seems to completely fix the issue on the videos I have tested so far. Is adding these dummies necessary in anyway ? Since you have two indices (frameIndex and PTS) already, what information do the dummies hold ?
I will test this edit on more videos and update the fork if it helps anyone else.
Many thanks Vlad ! Cheers --
@Al-Bas The dummies not hold any information, I introduced them in the first place to hide the complexity of video stream (missing frames, reordered frames) from the consumer.
P.S. also, my name is Vadim, not Vlad
@vadimkantorov Thanks, I tested it on 100 .mp4 contained videos pulled from different sources and all seems to be working well. Cheers --
@Al-Bas can you please explain more in details, how did you edit your code. is it just commenting this if section part https://github.com/vadimkantorov/mpegflow/blob/master/mpegflow.cpp#L310 ?
@Al-Bas Thank you. For me, it doesn't always solve the problem. I had two sequences: For the first one, I had some Invalid UE Golomb code but I get my motion vectors without skipping. Otherwise, with the dummies, I still have the skipping aaall the frames due to "std:bab_alloc" error. With another video sequence with high resolution, even skipping dummies doesn't solve it, my algorithm still skips some frames not all of them.
